We Keep Coding

python socket - how to complete/close the connection on the client side?

server.py:
json files from NVD are used here
import socket, json, random, threading, zipfile, requests, re, zipfile
from bs4 import BeautifulSoup
from zipfile import *
def listen_user(user):
for x in range(2018,2021,1):
filename = "nvdcve-1.1-" + str(x) + ".json"
print(filename)
with open(filename, 'rb') as file:
sendfile = file.read()
user.sendall(sendfile)
print('file sent' + str(x))
def start_server():
while True:
user_socket, address = server.accept()
print(f"User <{address[0]}> connected!")
users.append(user_socket)
listen_accepted_user = threading.Thread(
target=listen_user,
args=(user_socket,)
)
listen_accepted_user.start()
if __name__ == '__main__':
users = []
server = socket.socket(
socket.AF_INET,
socket.SOCK_STREAM,
)
server.bind(
("127.0.0.1", 100)
)
server.listen(5)
print('waiting for connection...')
start_server()
client.py
import socket, json, random
from threading import Thread
def start_client(client):
savefilename = str(random.randint(1,10)) + 'new.json'
print(savefilename)
with client,open(savefilename,'wb') as file:
while True:
recvfile = client.recv(4096)
if not recvfile:
print('1 client')
break
file.write(recvfile)
file.close()
print('2 client')
client.close()
if __name__ == '__main__':
client = socket.socket(
socket.AF_INET,
socket.SOCK_STREAM,
)
client.connect(
("127.0.0.1", 100)
)
start_client(client)
when I send files - they are sent almost in full, but the program does not reach the line "print ('1 client')" or "print ('2 client')"
and the *new file contains all lines except a few dozen of the last
please help - how to fix the code?

recvfile = client.recv(4096) is inside the while loop and it is continuously waiting for the next bytes to receive. The client doesn't know the files are sent, so it waits for the next 4096 bytes and doesn't exit the loop.
To let the client know that the file transfer is completed, you can send a message from the server.py which you can validate in the client and break the loop as shown below.
server.py
def listen_user(user):
for x in ["f.json","g.json"]:
filename = x
print(filename)
with open(filename, 'rb') as file:
sendfile = file.read()
user.sendall(sendfile)
print('file sent' + str(x))
user.send(b"Done")
Client.py
def start_client(client):
savefilename = str(random.randint(1,10)) + 'new.json'
print(savefilename)
with client,open(savefilename,'wb') as file:
while True:
recvfile = client.recv(4096)
if recvfile.decode("utf-8") =="Done":
print('1 client')
file.close()
break
file.write(recvfile)
print('2 client')
client.close()

The call client.recv(4096) means that you are waiting for 4096 bytes to be received, then doing something with those bytes. What's likely happening in this case is that you're writing out all of the bytes, minus those that don't quite fill up the buffer at the end. This leaves the client waiting with a buffer with space that is doesn't think it is ready to write out yet.
I'm guessing that you're assuming that client.recv() will return an empty string once you've gotten all the data; this is not the case based on your code. If you want the client to be able to terminate the connection, you're going to need to send some kind of control sequence or try to otherwise assess the bytes received from the server to determined when it's time to close the connection. If you do this, you will probably want to set bufsize when calling client.recv() to 1, and instead use some other method to buffer before you write to a file.
For instance, since you're sending JSON data, you could concatenate the bytes to a variable and then repeatedly try to parse JSON. Once you have managed to successfully parse JSON, you can terminate the connection on the client side (though this would mean you have to open a new connection per file you're sending).
However, that raises the question: why do you need to close from the client side? Usually the server will just close the connection once it is done sending all of the relevant data.

Transfer contents of a folder over network by python

I am facing a problem writing a program to send contents of a folder over the network by using Python. There are a lot of examples out there, all the examples I found are assuming the receiver side knew name of the file he want to receive. The program I am trying to do assuming that the receiver side agree to receive a files and there is no need to request a file by its name from the server. Once the connection established between the server and the client, the server start send all files inside particular folder to the client. Here is a image to show more explanation:example here
Here are some programs that do client server but they send one file and assume the receiver side knew files names, so the client should request a file by its name in order to receive it.
Note: I apologies for English grammar mistakes.
https://www.youtube.com/watch?v=LJTaPaFGmM4
http://www.bogotobogo.com/python/python_network_programming_server_client_file_transfer.php
python socket file transfer
Here is best example I found:
Server side:
import sys
import socket
import os
workingdir = "/home/SomeFilesFolder"
host = ''
skServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skServer.bind((host, 1000))
skServer.listen(10)
print "Server Active"
bFileFound = 0
while True:
Content, Address = skServer.accept()
print Address
sFileName = Content.recv(1024)
for file in os.listdir(workingdir):
if file == sFileName:
bFileFound = 1
break
if bFileFound == 0:
print sFileName + " Not Found On Server"
else:
print sFileName + " File Found"
fUploadFile = open("files/" + sFileName, "rb")
sRead = fUploadFile.read(1024)
while sRead:
Content.send(sRead)
sRead = fUploadFile.read(1024)
print "Sending Completed"
break
Content.close()
skServer.close()
Client side:
import sys
import socket
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("ip address", 1000))
sFileName = raw_input("Enter Filename to download from server : ")
sData = "Temp"
while True:
skClient.send(sFileName)
sData = skClient.recv(1024)
fDownloadFile = open(sFileName, "wb")
while sData:
fDownloadFile.write(sData)
sData = skClient.recv(1024)
print "Download Completed"
break
skClient.close()
if there is a way to eliminate this statement from the client side:
sFileName = raw_input("Enter Filename to download from server : ")
and make the server side send all files one by one without waiting for the client to pick a file.

Here's an example that recursively sends anything in the "server" subdirectory to a client. The client will save anything received in a "client" subdirectory. The server sends for each file:
The path and filename relative to the server subdirectory, UTF-8-encoded and terminated with a newline.
The file size in decimal as a UTF-8-encoded string terminated with a newline.
Exactly "file size" bytes of file data.
When all files are transmitted the server closes the connection.
server.py
from socket import *
import os
CHUNKSIZE = 1_000_000
sock = socket()
sock.bind(('',5000))
sock.listen(1)
while True:
print('Waiting for a client...')
client,address = sock.accept()
print(f'Client joined from {address}')
with client:
for path,dirs,files in os.walk('server'):
for file in files:
filename = os.path.join(path,file)
relpath = os.path.relpath(filename,'server')
filesize = os.path.getsize(filename)
print(f'Sending {relpath}')
with open(filename,'rb') as f:
client.sendall(relpath.encode() + b'\n')
client.sendall(str(filesize).encode() + b'\n')
# Send the file in chunks so large files can be handled.
while True:
data = f.read(CHUNKSIZE)
if not data: break
client.sendall(data)
print('Done.')
The client creates a "client" subdirectory and connects to the server. Until the server closes the connection, the client receives the path and filename, the file size, and the file contents and creates the file in the path under the "client" subdirectory.
client.py
from socket import *
import os
CHUNKSIZE = 1_000_000
# Make a directory for the received files.
os.makedirs('client',exist_ok=True)
sock = socket()
sock.connect(('localhost',5000))
with sock,sock.makefile('rb') as clientfile:
while True:
raw = clientfile.readline()
if not raw: break # no more files, server closed connection.
filename = raw.strip().decode()
length = int(clientfile.readline())
print(f'Downloading {filename}...\n Expecting {length:,} bytes...',end='',flush=True)
path = os.path.join('client',filename)
os.makedirs(os.path.dirname(path),exist_ok=True)
# Read the data in chunks so it can handle large files.
with open(path,'wb') as f:
while length:
chunk = min(length,CHUNKSIZE)
data = clientfile.read(chunk)
if not data: break
f.write(data)
length -= len(data)
else: # only runs if while doesn't break and length==0
print('Complete')
continue
# socket was closed early.
print('Incomplete')
break
Put any number of files and subdirectories under a "server" subdirectory in the same directory as server.py. Run the server, then in another terminal run client.py. A client subdirectory will be created and the files under "server" copied to it.

So... I've decided I've posted enough in comments and I might as well post a real answer. I see three ways to do this: push, pull, and indexing.
Push
Recall the HTTP protocol. The client asks for a file, the server locates it, and sends it. So get a list of all the files in a directory and send them all together. Better yet, tar them all together, zip them with some compression algorithm, and send that ONE file. This method is actually pretty much industry standard among Linux users.
Pull
I identifed this in the comments, but it works like this:
Client asks for directory
Server returns a text file containing the names of all the files.
Client asks for each file.
Index
This technique is the least mutable of the three. Keep an index of all the files in the directory, named INDEX.xml (funny enough, you could model the entire directory tree in xml.) your client will request the xml file, then walk the tree requesting other files.

you need to send os.listdir() by using json.dumps() and encode it as utf-8
at client side you need to decode and use json.loads() so that list will be transfer to client
place sData = skClient.recv(1024) before sFileName = raw_input("Enter Filename to download from server : ") so that the server file list can be display
you can find at here its a interesting tool
https://github.com/manoharkakumani/mano

Python socket programming for Webserver

I have created a proxy server that receives requests, searches for the requested file in its cache. If available it returns the cached file. If file is not available then it will ask the actual server, gets it, stores it in the cache and returns the file to the client.
Following is the code:
from socket import *
import sys
if len(sys.argv) <= 1:
print 'Usage : "python ProxyServer.py server_ip"\n[server_ip : It is the IP Address Of Proxy Server'
sys.exit(2)
# Create a server socket, bind it to a port and start listening
tcpSerSock = socket(AF_INET, SOCK_STREAM)
tcpSerSock.bind((sys.argv[1], 8888))
tcpSerSock.listen(100)
while 1:
# Strat receiving data from the client
print 'Ready to serve...'
tcpCliSock, addr = tcpSerSock.accept()
print 'Received a connection from:', addr
message = tcpCliSock.recv(1024)
print message
# Extract the filename from the given message
print message.split()[1]
filename = message.split()[1].partition("/")[2]
print filename
fileExist = "false"
filetouse = "/" + filename
print filetouse
try:
# Check wether the file exist in the cache
f = open(filetouse[1:], "r")
outputdata = f.readlines()
fileExist = "true"
# ProxyServer finds a cache hit and generates a response message
tcpCliSock.send("HTTP/1.0 200 OK\r\n")
tcpCliSock.send("Content-Type:text/html\r\n")
for i in range(0, len(outputdata)):
tcpCliSock.send(outputdata[i])
print 'Read from cache'
# Error handling for file not found in cache
except IOError:
if fileExist == "false":
# Create a socket on the proxyserver
c = socket(AF_INET, SOCK_STREAM)
hostn = filename.replace("www.","",1)
print hostn
try:
# Connect to the socket to port 80
c.connect((hostn, 80))
# Create a temporary file on this socket and ask port 80 for the file requested by the client
fileobj = c.makefile('r', 0)
fileobj.write("GET "+"http://" + filename + " HTTP/1.0\n\n")
# Read the response into buffer
buff = fileobj.readlines()
# Create a new file in the cache for the requested file. Also send the response in the buffer to client socket and the corresponding file in the cache
tmpFile = open("./" + filename,"wb")
for line in buff:
tmpFile.write(line);
tcpCliSock.send(line);
except:
print "Illegal request"
else:
# HTTP response message for file not found
tcpCliSock.send("HTTP/1.0 404 sendErrorErrorError\r\n")
tcpCliSock.send("Content-Type:text/html\r\n")
tcpCliSock.send("\r\n")
# Close the client and the server sockets
tcpCliSock.close()
tcpSerSock.close()
But for every file I request I only get an "illegal request" message printed. There seems to be an issue that the proxy server actually is not able to retrieve the requested file by the client. Can someone tell me where I can improve the code.
This is the first time I am coding in Python so please mention any minor errors.

Your request is illegal. For normal http servers, GET must not contain a URL, but only the path. The rest of your proxy contains also many errors. You probably want to use sendall everywhere you use send. recv can receive less that one message, so you have to handle this case also.
Why do you use the strings "true" and "false" instead of True and False?
There is a security hole, as you can read any file on your computer through your proxy. Reading binary files won't work. You don't close opened files.

copying same file name from client to server using tcp protocol with same size of file

This is the client and server program where a client sends a file to server to save in the server. There is a issuse in that same file name is not getting copied on the server with same file size
Please help me in this
Client program
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("localhost",9999))
path=raw_input("Please enter the complete PATH of your file : ")
f=open (path, "rb")
l = f.read(256)
while (l):
s.sendall(l)
l = f.read(10000)
s.close()
Server Program
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("localhost",9999))
s.listen(10)
while True:
s, address = s.accept()
print address
i=1
f = open( str(i),'wb') #open in binary
#i=i+1
while (True):
l=s.recv(256)
#while (l):
f.write(l)
l=s.recv(256)
print 'File recieve succesfully'
f.close()
#sc.close()
s.close()
Thanks in advance

Start by walking through the code and thinking about what the client knows about the data it is sending and what the server knows about the data it is receiving. You will have to send 2 types of messages: the data and the filename. How you do that is up to you.
Without over-thinking it, maybe try writing the filename first (followed by a newline or special character) then send the file data. On the server side accept the connection, read in data until you find a newline character (that's the filename), then receive the rest of the data and write it to the file.
Also, the server code you've provided doesn't work, at least I don't think, since you never break out of your while True loops.

How to send file from Client to Server and save it with same filename?

I'm trying to send file from client to server in python. It is sending without any problem but I want to save that received file with same file name. I'm not getting idea how to save that file with same file name as it is sent from Client to Server.The code I've wrote for this is :
Client Code
import socket, os, shutil
from stat import ST_SIZE
HOST=raw_input("Please enter IP-address : ")
PORT=int(raw_input("Please enter PORT Number : "))
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST,PORT))
if s.recv(8)!='READY':
raw_input('Unable to connect \n\n Press any key to exit ...')
s.close()
exit()
path=raw_input("Please enter the complete PATH of your file : ")
f=open(path,'rb')
fsize=os.stat(f.name)[ST_SIZE]
s.sendall(str(fsize).zfill(8))
sfile = s.makefile("wb")
shutil.copyfileobj(f, sfile)
sfile.close()
s.close()
f.close()
Server Code
import socket
import shutil
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = ''
PORT = 23240
s.bind((HOST, PORT))
s.listen(3)
conn, addr = s.accept()
print 'conn at address',addr
conn.sendall('READY')
i=1
f = open(r'file_'+ str(i)+".txt",'wb')
i=i+1
print 'File size',fsize
sfile = conn.makefile("rb")
shutil.copyfileobj(sfile, f)
sfile.close()
f.write(conn.recv(fsize))
f.close()
conn.close()
s.close()

Your code is not very robust. recv(cnt) delivers up to cnt bytes of data, or less. So it's not sure, you read the whole file. It is even not sure, you get the "READY" in one recv. Instead, you have to use something like that:
def recv_all(sock, bufsize):
result = ''
while bufsize>0:
data = sock.recv(min(bufsize, 4096))
if not data:
raise IOError("Socket closed")
result += data
bufsize -= len(data)
return result
If you want to know the filename at the server, you also have to transfer it to the server, too. By the way, "READY" has 5 characters, not 8.