I am facing a problem writing a program to send contents of a folder over the network by using Python. There are a lot of examples out there, all the examples I found are assuming the receiver side knew name of the file he want to receive. The program I am trying to do assuming that the receiver side agree to receive a files and there is no need to request a file by its name from the server. Once the connection established between the server and the client, the server start send all files inside particular folder to the client. Here is a image to show more explanation:example here
Here are some programs that do client server but they send one file and assume the receiver side knew files names, so the client should request a file by its name in order to receive it.
Note: I apologies for English grammar mistakes.
https://www.youtube.com/watch?v=LJTaPaFGmM4
http://www.bogotobogo.com/python/python_network_programming_server_client_file_transfer.php
python socket file transfer
Here is best example I found:
Server side:
import sys
import socket
import os
workingdir = "/home/SomeFilesFolder"
host = ''
skServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skServer.bind((host, 1000))
skServer.listen(10)
print "Server Active"
bFileFound = 0
while True:
Content, Address = skServer.accept()
print Address
sFileName = Content.recv(1024)
for file in os.listdir(workingdir):
if file == sFileName:
bFileFound = 1
break
if bFileFound == 0:
print sFileName + " Not Found On Server"
else:
print sFileName + " File Found"
fUploadFile = open("files/" + sFileName, "rb")
sRead = fUploadFile.read(1024)
while sRead:
Content.send(sRead)
sRead = fUploadFile.read(1024)
print "Sending Completed"
break
Content.close()
skServer.close()
Client side:
import sys
import socket
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("ip address", 1000))
sFileName = raw_input("Enter Filename to download from server : ")
sData = "Temp"
while True:
skClient.send(sFileName)
sData = skClient.recv(1024)
fDownloadFile = open(sFileName, "wb")
while sData:
fDownloadFile.write(sData)
sData = skClient.recv(1024)
print "Download Completed"
break
skClient.close()
if there is a way to eliminate this statement from the client side:
sFileName = raw_input("Enter Filename to download from server : ")
and make the server side send all files one by one without waiting for the client to pick a file.
Here's an example that recursively sends anything in the "server" subdirectory to a client. The client will save anything received in a "client" subdirectory. The server sends for each file:
The path and filename relative to the server subdirectory, UTF-8-encoded and terminated with a newline.
The file size in decimal as a UTF-8-encoded string terminated with a newline.
Exactly "file size" bytes of file data.
When all files are transmitted the server closes the connection.
server.py
from socket import *
import os
CHUNKSIZE = 1_000_000
sock = socket()
sock.bind(('',5000))
sock.listen(1)
while True:
print('Waiting for a client...')
client,address = sock.accept()
print(f'Client joined from {address}')
with client:
for path,dirs,files in os.walk('server'):
for file in files:
filename = os.path.join(path,file)
relpath = os.path.relpath(filename,'server')
filesize = os.path.getsize(filename)
print(f'Sending {relpath}')
with open(filename,'rb') as f:
client.sendall(relpath.encode() + b'\n')
client.sendall(str(filesize).encode() + b'\n')
# Send the file in chunks so large files can be handled.
while True:
data = f.read(CHUNKSIZE)
if not data: break
client.sendall(data)
print('Done.')
The client creates a "client" subdirectory and connects to the server. Until the server closes the connection, the client receives the path and filename, the file size, and the file contents and creates the file in the path under the "client" subdirectory.
client.py
from socket import *
import os
CHUNKSIZE = 1_000_000
# Make a directory for the received files.
os.makedirs('client',exist_ok=True)
sock = socket()
sock.connect(('localhost',5000))
with sock,sock.makefile('rb') as clientfile:
while True:
raw = clientfile.readline()
if not raw: break # no more files, server closed connection.
filename = raw.strip().decode()
length = int(clientfile.readline())
print(f'Downloading {filename}...\n Expecting {length:,} bytes...',end='',flush=True)
path = os.path.join('client',filename)
os.makedirs(os.path.dirname(path),exist_ok=True)
# Read the data in chunks so it can handle large files.
with open(path,'wb') as f:
while length:
chunk = min(length,CHUNKSIZE)
data = clientfile.read(chunk)
if not data: break
f.write(data)
length -= len(data)
else: # only runs if while doesn't break and length==0
print('Complete')
continue
# socket was closed early.
print('Incomplete')
break
Put any number of files and subdirectories under a "server" subdirectory in the same directory as server.py. Run the server, then in another terminal run client.py. A client subdirectory will be created and the files under "server" copied to it.
So... I've decided I've posted enough in comments and I might as well post a real answer. I see three ways to do this: push, pull, and indexing.
Push
Recall the HTTP protocol. The client asks for a file, the server locates it, and sends it. So get a list of all the files in a directory and send them all together. Better yet, tar them all together, zip them with some compression algorithm, and send that ONE file. This method is actually pretty much industry standard among Linux users.
Pull
I identifed this in the comments, but it works like this:
Client asks for directory
Server returns a text file containing the names of all the files.
Client asks for each file.
Index
This technique is the least mutable of the three. Keep an index of all the files in the directory, named INDEX.xml (funny enough, you could model the entire directory tree in xml.) your client will request the xml file, then walk the tree requesting other files.
you need to send os.listdir() by using json.dumps() and encode it as utf-8
at client side you need to decode and use json.loads() so that list will be transfer to client
place sData = skClient.recv(1024) before sFileName = raw_input("Enter Filename to download from server : ") so that the server file list can be display
you can find at here its a interesting tool
https://github.com/manoharkakumani/mano
Related
I've set up a python client and server with socket in Python, that allows the server to send text to the client and I've been trying to extend it so that images can be sent to the client.
Server code:
import socket
#setup and bind server socket
s_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)#setup socket
s_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)#reuses same port (allows reconnection)
s_socket.bind(('192.168.178.52', 9001))
s_socket.listen(1)
#connects and prints clients data and send message
clientsocket, address = s_socket.accept()
print('Connection from {}'.format(address))
clientsocket.send(bytes('Welcome to the server', 'utf-8'))
#Loops for server to sent text data to client
while True:
m = input('Enter')
try:
file = open(m, 'rb')
b = file.read(2048)
clientsocket.send(b)
except:
clientsocket.send(bytes(m, 'utf-8'))
Client code:
import socket
import webbrowser
import os
import pyautogui
#setup and bind client socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.connect(('----------', 9001))#ip swapped for post
while True:
message = s.recv(2048)#recieves all messages sent with buffer size
if message:
txt = str(message)
with open('plswork.png', 'wb') as file:
file.write(message)
file.close()
The problem I'm having is that it will send the file over and create it perfectly fine, but only part of the image will load in when i open it (see image) I am pretty sure this is something to do with the buffer size however when I increase it, it wont recognise the file at all and I'll get an error trying to open the photo (preferably you would be able to send most photos). New to python sockets so any help would be appreciated!
(at the moment trying to send a pic of tux...)
https://i.stack.imgur.com/lBblq.png
I don't know the size of the file, but shouldn't you read the file until it is read completely and send data in chunks?
while True:
m = input('Enter')
try:
file = open(m, 'rb')
while True:
b = file.read(2048)
if not b:
break
clientsocket.send(b)
except:
clientsocket.send(bytes(m, 'utf-8'))
Client side had to be adapted as well.
Most network protocols add more information to simplify reception.
It could for example be a good idea, if you first send the number of bytes, that the welcome message contains, then the welcome message, then some indicator, that you will send a file, then some information, how many bytes you will send for the image and only then the bytes of the image
You will also find out, that it is complicated for the client to know what is the text message and what is part of the png file.
In fact if you would remove the input() command from the server and hard code a file name you might probably notice. that the welcome message and the png file could arrive combined at the client side. and it would have difficulties separating the two.
So to make your code robust, there's a lot of work to do.
This is the client and server program where a client sends a file to server to save in the server. There is a issuse in that same file name is not getting copied on the server with same file size
Please help me in this
Client program
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("localhost",9999))
path=raw_input("Please enter the complete PATH of your file : ")
f=open (path, "rb")
l = f.read(256)
while (l):
s.sendall(l)
l = f.read(10000)
s.close()
Server Program
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("localhost",9999))
s.listen(10)
while True:
s, address = s.accept()
print address
i=1
f = open( str(i),'wb') #open in binary
#i=i+1
while (True):
l=s.recv(256)
#while (l):
f.write(l)
l=s.recv(256)
print 'File recieve succesfully'
f.close()
#sc.close()
s.close()
Thanks in advance
Start by walking through the code and thinking about what the client knows about the data it is sending and what the server knows about the data it is receiving. You will have to send 2 types of messages: the data and the filename. How you do that is up to you.
Without over-thinking it, maybe try writing the filename first (followed by a newline or special character) then send the file data. On the server side accept the connection, read in data until you find a newline character (that's the filename), then receive the rest of the data and write it to the file.
Also, the server code you've provided doesn't work, at least I don't think, since you never break out of your while True loops.
I am trying to make a TCP port server in python. Here is my code so far:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('',4000))
sock.listen(1)
while 1:
client, address = sock.accept()
fileexists = client.RUNCOMMAND(does the file exist?)
if fileexists = 0:
client.close()
else if:
filedata = client.RUNCOMMAND(get the contents of the file)
if filedata = "abcdefgh":
client.send('Transfer file accepted.')
else:
client.send('Whoops, seems like you have a corrupted file!')
client.close()
I just have no idea how to run a command (RUNCOMMMAND) that would check if a file exists on the client.
Also, is there a way to check what operating system the client is on to run different commands (eg. linux will have a file finder different command than windows). And I totally understand if this isn't possible, but I am really hoping that there is a way to do this.
Thank you very much.
XMLRPC may help you.
XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP as a transport.
http://docs.python.org/2/library/xmlrpclib.html
You might want to look at the very handy bottle.py micro server. its great for small server tasks like this and you get the Http protocol on top of this. You just include one file with your code. http://bottlepy.org
here is code that will work from http://blah:8090/get/file or http://blah:8090/exists/file so to see the contents of /etc/hosts would be http://blah:8090/get/etc/hosts
#!/usr/bin/python
import bottle
import os.path
#bottle.route("/get/<filepath:path>")
def index(filepath):
filepath = "/" + filepath
print "getting", filepath
if not os.path.exists(filepath):
return "file not found"
print open(filepath).read() # prints file
return '<br>'.join(open(filepath).read().split("\n")) # prints file with <br> for browser readability
#bottle.route("/exists/<filepath:path>")
def test(filepath):
filepath = "/" + filepath
return str(os.path.exists(filepath))
bottle.run(host='0.0.0.0', port=8090, reloader=True)
the reloader option on the run method allows you to edit the code without manually restarting the server. Its quite handy.
I am working on a project where images are taken by my android phone and are stored in folders in my SD card. I am working on a python script that needs to periodically move the folders from the SD to a particular folder in my PC. The phone and the PC are connected over the mobile Hotspot.
I wrote a socket program with my PC as client and the mobile as server. But I am facing some problems with it. Though I could not move folders i tried moving images from the folder and i am facing the following problems
the image is copied in the form of an unknown file format.
i am unable to iterate the process at the server side to move all the images present in the folder
at the client I am not able to store it in the location i want. I try to send the folder name and the file name from the server before sending the image but the client is not taking that file name i sent, instead it searches a folder in that name.
I also have a problem with the size of the names sent to the client, how do i randomly change the size at the client side depending on the name sent from the server.
I need someones help to sort this problem.
Here is the client side code
import socket,os
import time
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(("192.168.43.1", 5005))
size = 1024
while True:
fln = client_socket.recv(size) # folder name
fn = client_socket.recv(size) # file name
fname = "E:\\Transfered\\"+fln+"\\"+fn
fp = open(fname,'w')
while True:
strng = client_socket.recv(1024)
if not strng:
break
fp.write(strng)
fp.close()
print "Data Received successfully"
exit()
#data = 'viewnior '+fname
#os.system(data)
My Server side code
import os
import sys,time
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("", 5005))
server_socket.listen(5)
client_socket, address = server_socket.accept()
print "Conencted to - ",address,"\n"
sb = '/mnt/sdcard/sb'
while True:
files = os.listdir(sb)
pages = 0;
while (files):
print '\nMaybe, pending work'
for au in files:
if (au.find('d')>-1): # searching for folder with a d
os.chdir(sb+'/'+au)
imgFiles = os.listdir(sb+'/'+au)
images = [img for img in imgFiles if img.endswith('.jpg')]
print '\n%s user done' %au
client_socket.send(au)
pages = 0;
#copies all .img files in the folder from server to client
for imgs in images:
print imgs
client_socket.send(imgs)
file_name = open(imgs,'r')
while True:
strng = file_name.readline(1024)
if not strng:
break
client_socket.send(strng)
file_name.close()
print "Data sent successfully"
os.remove(sb+'/'+au+'/'+imgs)
pages = pages + 1
time.sleep(1)
os.chdir(sb)
os.rmdir(au)
else:
time.sleep(2)
exit()
The problem seems to be using readline() on a binary file at the server side:
file_name = open(imgs,'rb')
while True:
strng = file_name.readline()
readline() reads data from file up to the next '\n' character. Using it on a binary file may result in reading a very long buffer! (Maybe even up to EOF). In that case, using socket.send() may fail to deliver the entire data, and the return value (=bytes transmitted) should be checked. The possibilities for fixing that is:
using socket.sendall() when sending, will send the entire buffer.
or, alternatively (may use both)
using file_name.read(1024) - which will bound the amount of data read each cycle.
I have modified the code enough to solve many of my problems now the only problem i want to solve is the image transfer. I opened the a .jpg file at the client and wrote the data into it. But the final file size is just 1kb less that the original size. I guess my work will be done if I sort that out. Can some one help me with it.
heres the code
server:
import os
import sys,time
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("", 5005))
server_socket.listen(5)
client_socket, address = server_socket.accept()
print "Conencted to - ",address,"\n"
sb = '/mnt/sdcard/sb'
while True:
files = os.listdir(sb)
pages = 0;
while (files):
print '\nMaybe, pending work'
for au in files:
if (au.find('d')>-1):
os.chdir(sb+'/'+au)
imgFiles = os.listdir(sb+'/'+au)
images = [img for img in imgFiles if img.endswith('.jpg')]
print '\n%s user done' %au
client_socket.send(au)
#copies all .img files in the folder from server to client
for imgs in images:
client_socket.send(imgs)
file_name = open(imgs,'rb')
while True:
strng = file_name.readline()
if not strng:
break
client_socket.send(strng)
file_name.close()
os.remove(sb+'/'+au+'/'+imgs)
print "Data sent successfully"
time.sleep(1)
os.chdir(sb)
os.rmdir(au)
else:
time.sleep(2)
exit()
Client:
import socket,os
import time
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(("192.168.43.1", 5005))
dst="E:\\Kiosk\\"
while True:
#folder name
fln = client_socket.recv(4)
os.chdir(dst);
dst = "E:\\Kiosk\\"+fln+"\\"
if not os.path.exists(dst): os.makedirs(dst)
fname = client_socket.recv(4)
os.chdir(dst)
fname = fname+'.jpg'
fp = open(fname,'wb')
# image
while True:
strng = client_socket.recv(1024)
if not strng:
break
fp.write(strng)
fp.close()
print "Data Received successfully"
exit()
#time.sleep(10)
#data = 'viewnior '+fname
#os.system(data)
I am trying to program compilation server which compiles a C program sent by client and returns an object file which can then be linked and executed at the client. Here are my client and server programs respectively
client.py:
# Compilation client program
import sys, socket, string
File = raw_input("Enter the file name:")
ssock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssock.connect(('localhost', 5000))
csock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
csock.connect(('localhost', 5001))
f = open(File, "rb")
data = f.read()
f.close()
ssock.send(File) #send filename
ssock.send(data) #send file
fd=raw_input("Enter a key to start recieving object file:")
data=csock.recv(1024) #receive status
if data=="sucess\n":
File=File.replace(".c",".o") #objectfile name
print "Object file, "+File+", recieved sucessfully"
else:
print "There are compilation errors in " + File
File="error.txt" #errorfile name
print "Errors are reported in the file error.txt"
fobj=open(File,"wb")
while 1:
data=ssock.recv(1024) # if any error in c sourcefile then error gets
# eported in errorfile "error.txt" else objectfile is
# returned from server
if not data:break
fobj.write(data)
fobj.close()
ssock.close()
csock.close()
server.py
#Compilation Server program
import subprocess
import socket, time, string, sys, urlparse, os
ssock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssock.bind(('', 5000))
ssock.listen(2)
print 'Server Listening on port 5000'
csock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
csock.bind(('', 5001))
csock.listen(2)
print 'Control server listening on port 5001'
client, claddr = ssock.accept()
controlsoc, caddr = csock.accept()
filename=client.recv(1024) #receive filename
print filename
############### This code is not working, i'm not getting the reason #######
############### I want to receive a file more than 1KB from client #######
f = open(filename,"wb") #receive file======
while 1:
data = client.recv(1024)
if not data: break
f.write(data)
f.close()
###############
###############
data="gcc -c " + filename + " 2> error.txt" #shell command to execute c source file
#report errors if any to error.txt
from subprocess import call
call(data,shell=True) #executes the above shell command
fil = filename.replace(".c",".o")
if (os.path.isfile(fil))== True: #test for existence of objectfile
data = "sucess\n" #no objectfile => error in compilation
filename = filename.replace(".c",".o")
else:
data = "unsucessful\n"
print data+"hi"
filename = "error.txt"
controlsoc.send(data)
f = open(filename,"rb")
data=f.read()
f.close()
print data
client.send(data)
client.close()
controlsoc.close()
I'm not able to recieve files of multiple KB. Is there any flaw in my code or how should i modify my code in order to achieve my objective of coding a compilation server.
Please help me with this regard..Thanks in advance
The problem here is you assume that ssock.send(File) will result in filename=client.recv(1024) reading exactly the filename and not more, but in fact the receiving side has no idea where the filename ends and you end up getting the file name and part of the data in the filename variable.
TCP connection is a bi-directional stream of bytes. It doesn't know about boundaries of your messages. One send might correspond to more then one recv on the other side (and the other way around). You need an application-level protocol on top of raw TCP.
The easiest in your case would be to send a text line in the form file-size file-name\n as a header. This way your server would be able to not only separate header from file data (via newline) but also know how many bytes of file content to expect, and reuse same TCP connection for multiple files.