We Keep Coding

How to send files over Internet not LAN?

So, me and my friend just wanted to try creating our own personal chat application in python I started by trying to make a CLI application first, I googled a few things but I am unable to find out how to send files over Internet . There are a lot of ways to send it over LAN but me and my friend are on different networks so that won't work. I can't seem to find a way to do it over the internet. I am new to networking so please pardon if I have made any mistake. Here's the code I have to transfer files over LAN:
client.py
import socket
from threading import Thread
from datetime import datetime
# server's IP address
# if the server is not on this machine,
# put the private (network) IP address (e.g 192.168.1.2)
SERVER_HOST = input("Server IP : ")
SERVER_PORT = 5002 # server's port
separator_token = "<SEP>" # we will use this to separate the client name & message
# initialize TCP socket
s = socket.socket()
print(f"[*] Connecting to {SERVER_HOST}:{SERVER_PORT}...")
# connect to the server
s.connect((SERVER_HOST, SERVER_PORT))
print("[+] Connected.")
# prompt the client for a name
name = input("Enter your name: ")
def listen_for_messages():
while True:
message = s.recv(1024).decode()
print("\n" + message)
# make a thread that listens for messages to this client & print them
t = Thread(target=listen_for_messages)
# make the thread daemon so it ends whenever the main thread ends
t.daemon = True
# start the thread
t.start()
while True:
# input message we want to send to the server
to_send = input()
# a way to exit the program
if to_send.lower() == 'q':
break
# add the datetime, name & the color of the sender
date_now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
to_send = f"[{date_now}] {name}{separator_token}{to_send}"
# finally, send the message
s.send(to_send.encode())
# close the socket
s.close()
server.py
import socket
from threading import Thread
# server's IP address
SERVER_HOST = "0.0.0.0"
SERVER_PORT = 5002 # port we want to use
separator_token = "<SEP>" # we will use this to separate the client name & message
# initialize list/set of all connected client's sockets
client_sockets = set()
# create a TCP socket
s = socket.socket()
# make the port as reusable port
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# bind the socket to the address we specified
s.bind((SERVER_HOST, SERVER_PORT))
# listen for upcoming connections
s.listen(5)
print(f"[*] Listening as {SERVER_HOST}:{SERVER_PORT}")
def listen_for_client(cs):
"""
This function keep listening for a message from `cs` socket
Whenever a message is received, broadcast it to all other connected clients
"""
while True:
try:
# keep listening for a message from `cs` socket
msg = cs.recv(1024).decode()
except Exception as e:
# client no longer connected
# remove it from the set
print(f"[!] Error: {e}")
client_sockets.remove(cs)
else:
# if we received a message, replace the <SEP>
# token with ": " for nice printing
msg = msg.replace(separator_token, ": ")
# iterate over all connected sockets
for client_socket in client_sockets:
# and send the message
client_socket.send(msg.encode())
while True:
# we keep listening for new connections all the time
client_socket, client_address = s.accept()
print(f"[+] {client_address} connected.")
# add the new connected client to connected sockets
client_sockets.add(client_socket)
# start a new thread that listens for each client's messages
t = Thread(target=listen_for_client, args=(client_socket,))
# make the thread daemon so it ends whenever the main thread ends
t.daemon = True
# start the thread
t.start()

Your code is fine. The issue is the network setup.
I assume you both have typical setups, so you are both behind NAT (network address translation). Other chat apps use a server that you can both connect to. If you put your friend's IP address you will try to open a connection to the router which likely won't answer.
One hack is to setup port forwarding on one of your routers. Say 5002, from your example, and forward all incoming traffic to the right local machine. This should get you running, but I agree it's not a easy thing to get working. An easier setup would be to use upnp which sets up the port forwarding rules automatically, but I don't know which package would do that (there must be one somewhere).
Another way would be to switch to ipv6, if your providers support it. Because of the increase in address space it's common to give out ipv6 ranges to customers, so that local machines can have public ipv6 addresses, which you can use to connect to. Here packages should be routed correctly, but firewalls might interfere.
Yet another hack, would be to bypass routers entirely and plug your machine directly into your uplink connection. This depends on your provider and network setup. This won't work if you have a router that also acts as a modem to connect over TV cable or telephone wires.

Diclaimer: this may not be a true answer (at least it is not the expected one), but it could not fit in a comment.
Luckily you gave some context about your real problem! Because it is neither a Python question nor a file transfer one. It is about how to exchange data between 2 private networks each protected with Network Address Translation by their ISP router. And there is no simple and direct answer, only possible workaround.
In fact the correct solution would be that either you or a friend of yours use a (virtual) machine in the cloud and installs the server there. That way the server woud have a true address and could easily be accessed from any of your clients. It has some drawback and pitfalls because you now have an application publicly accessible and always up, so security has to be considered...
The possible workarounds depends on your ISP routers. Some allow to configure a local address to be a DMZ that is an address that is accessible from both the internet and the rest of your LAN but only on certain ports. The rationale behind is to reduce as much as possible the attack surface: the machine in the DMZ will be vulnerable, but it should not contain sensitive data and should be easy to rebuild from scratch if compromised. But it only makes sense if you (or one of your friends) can dedicate a physical machine (raspberry or equivalent are not so expensive). Others allow to configure a special port to be automatically routed to an internal address. You lose the security benefit of a DMZ but no longer need to dedicate anything.
Unfortunately, as I initialy said, I have no code fix to propose because the probem is not there, nor any direct solution because the workarounds really depend on your ISP routers.

Is it possible to use the same network socket simultaneously for listen and connect with the same port?

I want to use socket to implement that the client/server can send and receive the files from each other. The client can send and receive the files from the server and vice versa for the server. Also, need to use the Tkinter module to complete the server and client GUI.
When the 2 GUIs are initialed, a thread is started to listen to the connection from the opposite end. That means the server has a thread for listening and accepting the connection from the client while receiving the file from the client, and at the client-side, the thread is used to listen and accept the connection from the server.
My confusion is that whether do I need 2 ports, one is used for sending file from the client to the server, another one is to receiving the files from the server? And do I need to create 2 sockets for sending and receiving files?
My current solution is using 2 ports and 2 sockets in the client-side. when sending the file, the client-side acts the client, while receiving file it acts the server. Correct?
To be brief, is it possible to use the same network socket simultaneously for listen and connect to with the same port? Or must I have two separate sockets, and if so, must the separate sockets also be bound to separate network ports? 
Here is a simplified code that explains my questions. Any advice/comments are appreciated. Thanks.
class client():
def __init__(self):
self.clientTcpSock = None
self.initUI()
def initUI(self):
#create UI here
def createSocket(self):
s.clientTcpSock = socket()
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
s.bind((HOST, PORT_1)) #PORT_1 download file port
s.listen(5)
self.clientTcpSock = s
def sendFile(self):
# !!! Here creates a new TCP socket, or can i still use the same clientTcpSock??
s = socket()
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
s.connect((HOST, PORT_2)) # PORT_2 upload file port
s.send(b'hello world') # just for example
def receiveFile(self):
while True:
### receivefile here
self.clientTcpSock.close()
# call this function when init UI
def newTherad(self):
thread = threading.Thread(target=self.receiveFile, args=())
thread.setDaemon(True)
thread.start()

Python socket receiving corrupted information

I'm trying to understand how send and receive are working.
I was trying to send continuously data to a server and i noticed that the server would receive mixed bytes because i was sending to much data at a time. See my code:
Server:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("",1996))
server.listen(0)
c,d = server.accept()
while True:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
Client:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.connect(("192.168.1.4",1996))
while True:
data = 1
server.send( struct.pack("i", data) )
Then i change the while loops to this:
Server:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
c.send( str.encode("Server received your message. You now can continue
sending more data") )
Client:
data = 1
server.send( struct.pack("i", data) )
#Wait to secure the send.
server.recv(1024)
This is working. I'm making sure that the client won't send data before the
server already receive the previous send.
But what if i want to do the same for the server too? How can i make sure that the server will send bytes to the client in a safe way?
I already tried this and i notice that i created an infinity loop because(I used multi-threading in order to send and receive at the same time on the server):
client was sending some data and then waiting to get a signal from the server
that he can send again.
the server was getting some data then sending the signal and after that waiting for a signal from the user that he can send again.
But because the client was actually sending data again, the whole thing was going on again and this caused me an infinity talk-reply loop.
So what can i do to make a continuously conversation between two sockets without mixing the bytes together?

Your problem is caused by Nagle algorithm which works by combining a number of small outgoing messages, and sending them all at once as TCP is a stream protocol. You can enable TCP_NODELAY socket option by calling sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) to sent data as soon as possible, even if there is only a small amount of data. And on the receiver side, it isn't going to get one packet at a time either, you must implement message boundaries itself if you want "continuous conversation between two sockets without mixing the bytes together".

I need the server to send messages to all clients (Python, sockets)

This is my server program, how can it send the data received from each client to every other client?
import socket
import os
from threading import Thread
import thread
def listener(client, address):
print "Accepted connection from: ", address
while True:
data = client.recv(1024)
if not data:
break
else:
print repr(data)
client.send(data)
client.close()
host = socket.gethostname()
port = 10016
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host,port))
s.listen(3)
th = []
while True:
print "Server is listening for connections..."
client, address = s.accept()
th.append(Thread(target=listener, args = (client,address)).start())
s.close()

If you need to send a message to all clients, you need to keep a collection of all clients in some way. For example:
clients = set()
clients_lock = threading.Lock()
def listener(client, address):
print "Accepted connection from: ", address
with clients_lock:
clients.add(client)
try:
while True:
data = client.recv(1024)
if not data:
break
else:
print repr(data)
with clients_lock:
for c in clients:
c.sendall(data)
finally:
with clients_lock:
clients.remove(client)
client.close()
It would probably be clearer to factor parts of this out into separate functions, like a broadcast function that did all the sends.
Anyway, this is the simplest way to do it, but it has problems:
If one client has a slow connection, everyone else could bog down writing to it. And while they're blocking on their turn to write, they're not reading anything, so you could overflow the buffers and start disconnecting everyone.
If one client has an error, the client whose thread is writing to that client could get the exception, meaning you'll end up disconnecting the wrong user.
So, a better solution is to give each client a queue, and a writer thread servicing that queue, alongside the reader thread. (You can then extend this in all kinds of ways—put limits on the queue so that people stop trying to talk to someone who's too far behind, etc.)
As Anzel points out, there's a different way to design servers besides using a thread (or two) per client: using a reactor that multiplexes all of the clients' events.
Python 3.x has some great libraries for this built in, but 2.7 only has the clunky and out-of-date asyncore/asynchat and the low-level select.
As Anzel says, Python SocketServer: sending to multiple clients has an answer using asyncore, which is worth reading. But I wouldn't actually use that. If you want to write a reactor-based server in Python 2.x, I'd either use a better third-party framework like Twisted, or find or write a very simple one that sits directly on select.

How to avoid high cpu usage?

I created a zmq_forwarder.py that's run separately and it passes messages from the app to a sockJS connection, and i'm currently working on right now on how a flask app could receive a message from sockJS via zmq. i'm pasting the contents of my zmq_forwarder.py. im new to ZMQ and i dont know why everytime i run it, it uses 100% CPU load.
import zmq
# Prepare our context and sockets
context = zmq.Context()
receiver_from_server = context.socket(zmq.PULL)
receiver_from_server.bind("tcp://*:5561")
forwarder_to_server = context.socket(zmq.PUSH)
forwarder_to_server.bind("tcp://*:5562")
receiver_from_websocket = context.socket(zmq.PULL)
receiver_from_websocket.bind("tcp://*:5563")
forwarder_to_websocket = context.socket(zmq.PUSH)
forwarder_to_websocket.bind("tcp://*:5564")
# Process messages from both sockets
# We prioritize traffic from the server
while True:
# forward messages from the server
while True:
try:
message = receiver_from_server.recv(zmq.DONTWAIT)
except zmq.Again:
break
print "Received from server: ", message
forwarder_to_websocket.send_string(message)
# forward messages from the websocket
while True:
try:
message = receiver_from_websocket.recv(zmq.DONTWAIT)
except zmq.Again:
break
print "Received from websocket: ", message
forwarder_to_server.send_string(message)
as you can see, i've setup 4 sockets. the app connects to port 5561 to push data to zmq, and port 5562 to receive from zmq (although im still figuring out how to actually set it up to listen for messages sent by zmq). on the other hand, sockjs receives data from zmq on port 5564 and sends data to it on port 5563.
i've read the zmq.DONTWAIT makes receiving of message asynchronous and non-blocking so i added it.
is there a way to improve the code so that i dont overload the CPU? the goal is to be able to pass messages between the flask app and the websocket using zmq.

You are polling your two receiver sockets in a tight loop, without any blocking (zmq.DONTWAIT), which will inevitably max out the CPU.
Note that there is some support in ZMQ for polling multiple sockets in a single thread - see this answer. I think you can adjust the timeout in poller.poll(millis) so that your code only uses lots of CPU if there are lots of incoming messages, and idles otherwise.
Your other option is to use the ZMQ event loop to respond to incoming messages asynchronously, using callbacks. See the PyZMQ documentation on this topic, from which the following "echo" example is adapted:
# set up the socket, and a stream wrapped around the socket
s = ctx.socket(zmq.REP)
s.bind('tcp://localhost:12345')
stream = ZMQStream(s)
# Define a callback to handle incoming messages
def echo(msg):
# in this case, just echo the message back again
stream.send_multipart(msg)
# register the callback
stream.on_recv(echo)
# start the ioloop to start waiting for messages
ioloop.IOLoop.instance().start()