We Keep Coding

How to send files over Internet not LAN?

So, me and my friend just wanted to try creating our own personal chat application in python I started by trying to make a CLI application first, I googled a few things but I am unable to find out how to send files over Internet . There are a lot of ways to send it over LAN but me and my friend are on different networks so that won't work. I can't seem to find a way to do it over the internet. I am new to networking so please pardon if I have made any mistake. Here's the code I have to transfer files over LAN:
client.py
import socket
from threading import Thread
from datetime import datetime
# server's IP address
# if the server is not on this machine,
# put the private (network) IP address (e.g 192.168.1.2)
SERVER_HOST = input("Server IP : ")
SERVER_PORT = 5002 # server's port
separator_token = "<SEP>" # we will use this to separate the client name & message
# initialize TCP socket
s = socket.socket()
print(f"[*] Connecting to {SERVER_HOST}:{SERVER_PORT}...")
# connect to the server
s.connect((SERVER_HOST, SERVER_PORT))
print("[+] Connected.")
# prompt the client for a name
name = input("Enter your name: ")
def listen_for_messages():
while True:
message = s.recv(1024).decode()
print("\n" + message)
# make a thread that listens for messages to this client & print them
t = Thread(target=listen_for_messages)
# make the thread daemon so it ends whenever the main thread ends
t.daemon = True
# start the thread
t.start()
while True:
# input message we want to send to the server
to_send = input()
# a way to exit the program
if to_send.lower() == 'q':
break
# add the datetime, name & the color of the sender
date_now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
to_send = f"[{date_now}] {name}{separator_token}{to_send}"
# finally, send the message
s.send(to_send.encode())
# close the socket
s.close()
server.py
import socket
from threading import Thread
# server's IP address
SERVER_HOST = "0.0.0.0"
SERVER_PORT = 5002 # port we want to use
separator_token = "<SEP>" # we will use this to separate the client name & message
# initialize list/set of all connected client's sockets
client_sockets = set()
# create a TCP socket
s = socket.socket()
# make the port as reusable port
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# bind the socket to the address we specified
s.bind((SERVER_HOST, SERVER_PORT))
# listen for upcoming connections
s.listen(5)
print(f"[*] Listening as {SERVER_HOST}:{SERVER_PORT}")
def listen_for_client(cs):
"""
This function keep listening for a message from `cs` socket
Whenever a message is received, broadcast it to all other connected clients
"""
while True:
try:
# keep listening for a message from `cs` socket
msg = cs.recv(1024).decode()
except Exception as e:
# client no longer connected
# remove it from the set
print(f"[!] Error: {e}")
client_sockets.remove(cs)
else:
# if we received a message, replace the <SEP>
# token with ": " for nice printing
msg = msg.replace(separator_token, ": ")
# iterate over all connected sockets
for client_socket in client_sockets:
# and send the message
client_socket.send(msg.encode())
while True:
# we keep listening for new connections all the time
client_socket, client_address = s.accept()
print(f"[+] {client_address} connected.")
# add the new connected client to connected sockets
client_sockets.add(client_socket)
# start a new thread that listens for each client's messages
t = Thread(target=listen_for_client, args=(client_socket,))
# make the thread daemon so it ends whenever the main thread ends
t.daemon = True
# start the thread
t.start()

Your code is fine. The issue is the network setup.
I assume you both have typical setups, so you are both behind NAT (network address translation). Other chat apps use a server that you can both connect to. If you put your friend's IP address you will try to open a connection to the router which likely won't answer.
One hack is to setup port forwarding on one of your routers. Say 5002, from your example, and forward all incoming traffic to the right local machine. This should get you running, but I agree it's not a easy thing to get working. An easier setup would be to use upnp which sets up the port forwarding rules automatically, but I don't know which package would do that (there must be one somewhere).
Another way would be to switch to ipv6, if your providers support it. Because of the increase in address space it's common to give out ipv6 ranges to customers, so that local machines can have public ipv6 addresses, which you can use to connect to. Here packages should be routed correctly, but firewalls might interfere.
Yet another hack, would be to bypass routers entirely and plug your machine directly into your uplink connection. This depends on your provider and network setup. This won't work if you have a router that also acts as a modem to connect over TV cable or telephone wires.

Diclaimer: this may not be a true answer (at least it is not the expected one), but it could not fit in a comment.
Luckily you gave some context about your real problem! Because it is neither a Python question nor a file transfer one. It is about how to exchange data between 2 private networks each protected with Network Address Translation by their ISP router. And there is no simple and direct answer, only possible workaround.
In fact the correct solution would be that either you or a friend of yours use a (virtual) machine in the cloud and installs the server there. That way the server woud have a true address and could easily be accessed from any of your clients. It has some drawback and pitfalls because you now have an application publicly accessible and always up, so security has to be considered...
The possible workarounds depends on your ISP routers. Some allow to configure a local address to be a DMZ that is an address that is accessible from both the internet and the rest of your LAN but only on certain ports. The rationale behind is to reduce as much as possible the attack surface: the machine in the DMZ will be vulnerable, but it should not contain sensitive data and should be easy to rebuild from scratch if compromised. But it only makes sense if you (or one of your friends) can dedicate a physical machine (raspberry or equivalent are not so expensive). Others allow to configure a special port to be automatically routed to an internal address. You lose the security benefit of a DMZ but no longer need to dedicate anything.
Unfortunately, as I initialy said, I have no code fix to propose because the probem is not there, nor any direct solution because the workarounds really depend on your ISP routers.

How to make 2 clients connect each other directly, after having both connected a meeting-point server?

I'm writing a toy meeting-point/relay server listening on port 5555 for two clients "A" and "B".
It works like this: every byte received by the server from the firstly-connected client A will be sent to the secondly-connected client B, even if A and B don't know their respective IP:
A -----------> server <----------- B # they both connect the server first
A --"hello"--> server # A sends a message to server
server --"hello"--> B # the server sends the message to B
This code is currently working:
# server.py
import socket, time
from threading import Thread
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.bind(('', 5555))
socket.listen(5)
buf = ''
i = 0
def handler(client, i):
global buf
print 'Hello!', client, i
if i == 0: # client A, who sends data to server
while True:
req = client.recv(1000)
buf = str(req).strip() # removes end of line
print 'Received from Client A: %s' % buf
elif i == 1: # client B, who receives data sent to server by client A
while True:
if buf != '':
client.send(buf)
buf = ''
time.sleep(0.1)
while True: # very simple concurrency: accept new clients and create a Thread for each one
client, address = socket.accept()
print "{} connected".format(address)
Thread(target=handler, args=(client, i)).start()
i += 1
and you can test it by launching it on a server, and do two netcat connections to it: nc <SERVER_IP> 5555.
How can I then pass the information to the clients A and B that they can talk directly to each other without making the bytes transit via the server?
There are 2 cases:
General case, i.e. even if A and B are not in the same local network
Particular case where these two clients are in the same local network (example: using the same home router), this will be displayed on the server when the 2 clients will connect to the server on port 5555:
('203.0.113.0', 50340) connected # client A, router translated port to 50340
('203.0.113.0', 52750) connected # same public IP, client B, router translated port to 52750
Remark: a previous unsuccesful attempt here: UDP or TCP hole punching to connect two peers (each one behind a router)
and UDP hole punching with a third party

Since the server knows the addresses of both clients, it can send that information to them and so they would know each others adress. There are many ways the server can send this data - pickled, json-encoded, raw bytes. I think the best option is to convert the address to bytes, because the client will know exactly how many bytes to read: 4 for the IP (integer) and 2 for the port (unsigned short). We can convert an address to bytes and back with the functions below.
import socket
import struct
def addr_to_bytes(addr):
return socket.inet_aton(addr[0]) + struct.pack('H', addr[1])
def bytes_to_addr(addr):
return (socket.inet_ntoa(addr[:4]), struct.unpack('H', addr[4:])[0])
When the clients receive and decode the address, they no longer need the server, and they can establish a new connection between them.
Now we have two main otions, as far as I know.
One client acts as a server. This client would close the connection to the server and would start listening on the same port. The problem with this method is that it will only work if both clients are on the same local network, or if that port is open for incoming connections.
Hole punching. Both clients start sending and accepting data from each other simultaneously. The clients must accept data on the same address they used to connect to the rendezvous server, which is knwn to each other. That would punch a hole in the client's nat and the clients would be able to communicate directly even if they are on different networks. This proccess is expleined in detail in this article Peer-to-Peer Communication Across Network Address Translators, section 3.4 Peers Behind Different NATs.
A Python example for UDP Hole Punching:
Server:
import socket
def udp_server(addr):
soc = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
soc.bind(addr)
_, client_a = soc.recvfrom(0)
_, client_b = soc.recvfrom(0)
soc.sendto(addr_to_bytes(client_b), client_a)
soc.sendto(addr_to_bytes(client_a), client_b)
addr = ('0.0.0.0', 4000)
udp_server(addr)
Client:
import socket
from threading import Thread
def udp_client(server):
soc = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
soc.sendto(b'', server)
data, _ = soc.recvfrom(6)
peer = bytes_to_addr(data)
print('peer:', *peer)
Thread(target=soc.sendto, args=(b'hello', peer)).start()
data, addr = soc.recvfrom(1024)
print('{}:{} says {}'.format(*addr, data))
server_addr = ('server_ip', 4000) # the server's public address
udp_client(server_addr)
This code requires for the rendezvous server to have a port open (4000 in this case), and be accessible by both clients. The clients can be on the same or on different local networks. The code was tested on Windows and it works well, either with a local or a public IP.
I have experimented with TCP hole punching but I had limited success (sometimes it seems that it works, sometimes it doesn't). I can include the code if someone wants to experiment. The concept is more or less the same, both clients start sending and receiving simultaneously, and it is described in detail in Peer-to-Peer Communication Across Network Address Translators, section 4, TCP Hole Punching.
If both clients are on the same network, it will be much easier to communicate with each other. They would have to choose somehow which one will be a server, then they can create a normal server-client connection. The only problem here is that the clients must detect if they are on the same network. Again, the server can help with this problem, as it knows the public address of both clients. For example:
def tcp_server(addr):
soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
soc.bind(addr)
soc.listen()
client_a, addr_a = soc.accept()
client_b, addr_b = soc.accept()
client_a.send(addr_to_bytes(addr_b) + addr_to_bytes(addr_a))
client_b.send(addr_to_bytes(addr_a) + addr_to_bytes(addr_b))
def tcp_client(server):
soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
soc.connect(server)
data = soc.recv(12)
peer_addr = bytes_to_addr(data[:6])
my_addr = bytes_to_addr(data[6:])
if my_addr[0] == peer_addr[0]:
local_addr = (soc.getsockname()[0], peer_addr[1])
... connect to local address ...
Here the server sends two addresses to each client, the peer's public address and the client's own public address. The clients compare the two IPs, if they match then they must be on the same local network.

The accepted answer gives the solution. Here is some additional information in the case "Client A and Client B are in the same local network".
This situation can indeed be detected by the server if it notices that both clients have the same public IP.
Then the server can choose Client A as "local server", and Client B as "local client".
The server will then ask Client A for its "local network IP". Client A can find it with:
import socket
localip = socket.gethostbyname(socket.gethostname()) # example: 192.168.1.21
and then send it back to the server. The server will communicate this "local network IP" to Client B.
Then Client A will then run a "local server":
import socket
soc = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
soc.bind(('0.0.0.0', 4000))
data, client = soc.recvfrom(1024)
print("Connected client:", client)
print("Received message:", data)
soc.sendto(b"I am the server", client)
and Client B will run as a "local client":
import socket
soc = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server = ('192.168.1.21', 4000) # this "local network IP" has been sent Client A => server => Client B
soc.sendto("I am the client", server)
data, client = soc.recvfrom(1024)
print("Received message:", data)

Python server TCP server accessable from anywhere (via external IP, port forwarding is done)

I am new here, so please, don't be angry if I am stupid - but I don't know about it.
I would like to make a python TCP server, which can be accessed from anywhere via external (public) IP. I have done simple server (it works) in local network from this tutorial:
https://www.youtube.com/watch?v=XiVVYfgDolU
The client sends string and the server sends back that string but with uppercase.
Now I want to do the same but accessable from anywhere. I read a lot about it. I have Raspberry Pi, where I set up static IP address and I did the port forward (on port 42424). I am just looking for some tutorial, you can direct me anywhere - thats all I need. Or you can tell me how to do it step by step, but I know that it takes a lot of time to write answer. I tried googling, but I didn't find anything. And if I did, it was a person who didn't know what the external IP and the port forwarding is so the end of the conversation was: Learn what is port forwarding.
So please, give me some tips how to do it, or direct me somewhere. Thanks!
The code
Server:
import socket
def Main():
host = '10.0.0.140'
port = 42424
s = socket.socket()
s.bind((host, port))
s.listen(1)
c, addr = s.accept()
while True:
data = c.recv(1024)
if not data:
break
data = str(data).upper()
c.send(data)
c.close()
if __name__ == '__main__':
Main()
Client:
import socket
def Main():
host = '10.0.0.140'
port = 42424
s = socket.socket()
s.connect((host,port))
message = raw_input("->")
while message != 'q':
s.send(message)
data = s.recv(1024)
message = raw_input("->")
s.close()
if __name__ == '__main__':
Main()

When connecting to a server behind a NAT firewall/router, in addition to port forwarding, the client should be directed to the IP address of the router. As far as the client is concerned, the IP address of the router is the server. The router simply forwards the traffic according to the port forwarding rules.

Communicating with multiple clients using one TCP socket python

I am using TCP sockets to communicate between my server and clients. The server code and socket code are as below:
server:
from socket import *
HOST = 'xx.xx.xx.xx'
PORT = 1999
serversocket = socket(AF_INET,SOCK_STREAM)
serversocket.bind((HOST,PORT))
print 'bind success'
serversocket.listen(5)
print 'listening'
while True:
(clientsocket, address) = serversocket.accept()
print ("Got client request from",address)
#clientsocket.send('True')
data = clientsocket.recv(1024)
print data
clientsocket.send('True')
clientsocket.close()
client:
import socket
import sys
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect the socket to the port on the server given by the caller
server_address = ('xx.xx.xx.xx', 1999)
print >>sys.stderr, 'connecting to %s port %s' % server_address
sock.connect(server_address)
try:
message = 'This is the message. It will be repeated.'
print >>sys.stderr, 'sending'
for x in range (0,1):
name=raw_input ('what is ur name')
print type(name)
sock.send(name)
print sock.recv(1024)
finally:
sock.close()
I am able to communicate with the server from client and able to send and receive data. But the problem I am facing is that I am not able to send and receive data continuously from the server. I have to restart my client code on my laptop to send and receive data again from the server. The way the above client code is working is that when I give a keyboard input, then the socket sends data to server and server responds back. But in the client code, in the for loop if I do two iterations, for the second iteration the data I enter from keyboard is not reaching server. I need to restart my client code to send data again. How do I fix this ?
Also, when once client is connected to the server, the other cannot connect to the server. Any ideas on how to do this ?

You need to design and implement a protocol that specifies what each side is supposed to do and then implement that protocol. You're expecting it to work by magic.
For example:
data = clientsocket.recv(1024)
I suspect you are expecting this to receive a "message". But TCP has no notion of messages. If you need to send and receive messages, you need to define precisely what a "message" is for your protocol and write code to send and receive them.
It may be helpful to look at the specifications for other protocols that use TCP such as HTTP, FTP, or IRC. It really is worth the time to write out a specification of your protocol before you write any code. It will save a lot of pain.

Python: Socket Programming: accept() and connect calls

I have been self-learning python since few months now , and finally learning Socket programming. As an text book exercise, I am supposed to design a half-duplex chat system . Below is the code. The first request and response are just fine , but everytime I try sending a second message from client, the server seems to be hanging. The program is TCP based.
I am suspecting that since ss.accept() is being called everytime a new message has to be sent, a new thread is being created but since I have made only 1 call to sc.connect() from client , may be my new connection at the server end is hanging there for infinite time.
As a trail : I called ss.accept() outside the while loop, ie making only 1 connection and listening to data over and over on while loop, the conversations works just fine
Can someone please have a look a the code and help me understand where exactly is the issue.
Since, I am learning, I have not moved to twisted yet. I want to learn all the basics first before I move to frameworks.
!bin/usr/env python
import socket, sys
HOST =''
PORT = 1060
ADDR =(HOST,PORT)
def userinput(sock):
usermessage = input('>')
sock.sendall(str(len(usermessage)))
return usermessage
def server():
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind(ADDR)
s.listen(1)
print 'the server is listening at',s.getsockname()
while True:
ss,sockname = s.accept()
#listen to determine the bytes sent by client
msglen = ss.recv(4096)
#accept the complete message
msg = ss.recv(int(msglen))
print 'client:', repr(msg)
servermsg = userinput(ss)
ss.sendall(servermsg)
print " ---------------"
ss.close()
def client():
sc = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sc.connect(ADDR)
while True:
message = userinput(sc)
sc.sendall(message)
replylen = sc.recv(4096)
reply = sc.recv(int(replylen))
print "reply:",reply
print "----------------"
sc.close()
if sys.argv[1:] == ['server']:
server()
elif sys.argv[1:] == ['client']:
client()
else:
print >> sys.stderr,'usage:tcp_2_7.py server|client[host]'

Your trial - accepting once and then receiving multiple messages - is how you should do this. Calling accept is waiting for a new connection - you don't need to do this every time you want to send or receive a message, just as you don't want to call connect every time you want to send or receive.
Think of it this way:
When you connect to a chat server, do you connect, send a message, then disconnect immediately? No - you have a constant open connection which messages are sent through, and the connection is only closed at the end of a chat session.
From the docs on accept:
socket.accept()
Accept a connection. The socket must be bound to an
address and listening for connections. The return value is a pair
(conn, address) where conn is a new socket object usable to send and
receive data on the connection, and address is the address bound to
the socket on the other end of the connection.