We Keep Coding

Sending png files over python sockets

I've set up a python client and server with socket in Python, that allows the server to send text to the client and I've been trying to extend it so that images can be sent to the client.
Server code:
import socket
#setup and bind server socket
s_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)#setup socket
s_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)#reuses same port (allows reconnection)
s_socket.bind(('192.168.178.52', 9001))
s_socket.listen(1)
#connects and prints clients data and send message
clientsocket, address = s_socket.accept()
print('Connection from {}'.format(address))
clientsocket.send(bytes('Welcome to the server', 'utf-8'))
#Loops for server to sent text data to client
while True:
m = input('Enter')
try:
file = open(m, 'rb')
b = file.read(2048)
clientsocket.send(b)
except:
clientsocket.send(bytes(m, 'utf-8'))
Client code:
import socket
import webbrowser
import os
import pyautogui
#setup and bind client socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.connect(('----------', 9001))#ip swapped for post
while True:
message = s.recv(2048)#recieves all messages sent with buffer size
if message:
txt = str(message)
with open('plswork.png', 'wb') as file:
file.write(message)
file.close()
The problem I'm having is that it will send the file over and create it perfectly fine, but only part of the image will load in when i open it (see image) I am pretty sure this is something to do with the buffer size however when I increase it, it wont recognise the file at all and I'll get an error trying to open the photo (preferably you would be able to send most photos). New to python sockets so any help would be appreciated!
(at the moment trying to send a pic of tux...)
https://i.stack.imgur.com/lBblq.png

I don't know the size of the file, but shouldn't you read the file until it is read completely and send data in chunks?
while True:
m = input('Enter')
try:
file = open(m, 'rb')
while True:
b = file.read(2048)
if not b:
break
clientsocket.send(b)
except:
clientsocket.send(bytes(m, 'utf-8'))
Client side had to be adapted as well.
Most network protocols add more information to simplify reception.
It could for example be a good idea, if you first send the number of bytes, that the welcome message contains, then the welcome message, then some indicator, that you will send a file, then some information, how many bytes you will send for the image and only then the bytes of the image
You will also find out, that it is complicated for the client to know what is the text message and what is part of the png file.
In fact if you would remove the input() command from the server and hard code a file name you might probably notice. that the welcome message and the png file could arrive combined at the client side. and it would have difficulties separating the two.
So to make your code robust, there's a lot of work to do.

How to send and receive webcam stream using tcp sockets in Python?

I am trying to recreate this project. What I have is a server (my computer), and a client (my raspberry pi). What I am doing differently than the original project is that I am trying to use a simple webcam instead of a raspberry pi camera to stream images from my rpi to the server. I know that I must:
Get opencv image frames from the camera.
Convert a frame (which is a numpy array) to bytes.
Transfer the bytes from the client to the server.
Convert the bytes back into frames and view.
Examples would be appreciated.
self_driver.py
import SocketServer
import threading
import numpy as np
import cv2
import sys
ultrasonic_data = None
#BaseRequestHandler is used to process incoming requests
class UltrasonicHandler(SocketServer.BaseRequestHandler):
data = " "
def handle(self):
while self.data:
self.data = self.request.recv(1024)
ultrasonic_data = float(self.data.split('.')[0])
print(ultrasonic_data)
#VideoStreamHandler uses streams which are file-like objects for communication
class VideoStreamHandler(SocketServer.StreamRequestHandler):
def handle(self):
stream_bytes = b''
try:
stream_bytes += self.rfile.read(1024)
image = np.frombuffer(stream_bytes, dtype="B")
print(image.shape)
cv2.imshow('F', image)
cv2.waitKey(0)
finally:
cv2.destroyAllWindows()
sys.exit()
class Self_Driver_Server:
def __init__(self, host, portUS, portCam):
self.host = host
self.portUS = portUS
self.portCam = portCam
def startUltrasonicServer(self):
# Create the Ultrasonic server, binding to localhost on port 50001
server = SocketServer.TCPServer((self.host, self.portUS), UltrasonicHandler)
server.serve_forever()
def startVideoServer(self):
# Create the video server, binding to localhost on port 50002
server = SocketServer.TCPServer((self.host, self.portCam), VideoStreamHandler)
server.serve_forever()
def start(self):
ultrasonic_thread = threading.Thread(target=self.startUltrasonicServer)
ultrasonic_thread.daemon = True
ultrasonic_thread.start()
self.startVideoServer()
if __name__ == "__main__":
#From SocketServer documentation
HOST, PORTUS, PORTCAM = '192.168.0.18', 50001, 50002
sdc = Self_Driver_Server(HOST, PORTUS, PORTCAM)
sdc.start()
video_client.py
import socket
import time
import cv2
client_sock = socket.socket()
client_sock.connect(('192.168.0.18', 50002))
#We are going to 'write' to a file in 'binary' mode
conn = client_sock.makefile('wb')
try:
cap = cv2.VideoCapture(0)
cap.set(cv2.cv.CV_CAP_PROP_FRAME_WIDTH,320)
cap.set(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT,240)
start = time.time()
while(cap.isOpened()):
conn.flush()
ret, frame = cap.read()
byteImage = frame.tobytes()
conn.write(byteImage)
finally:
finish = time.time()
cap.release()
client_sock.close()
conn.close()

You can't just display every received buffer of 1-1024 bytes as an image; you have to concatenate them up and only display an image when your buffer is complete.
If you know, out of band, that your images are going to be a fixed number of bytes, you can do something like this:
IMAGE_SIZE = 320*240*3
def handle(self):
stream_bytes = b''
try:
stream_bytes += self.rfile.read(1024)
while len(stream_bytes) >= IMAGE_SIZE:
image = np.frombuffer(stream_bytes[:IMAGE_SIZE], dtype="B")
stream_bytes = stream_bytes[IMAGE_SIZE:]
print(image.shape)
cv2.imshow('F', image)
cv2.waitKey(0)
finally:
cv2.destroyAllWindows()
sys.exit()
If you don't know that, you have to add some kind of framing protocol, like sending the frame size as a uint32 before each frame, so the server can know how many bytes to received for each frame.
Next, if you're just sending the raw bytes, without any dtype or shape or order information, you need to embed the dtype and shape information into the server. If you know it's supposed to be, say, bytes in C order in a particular shape, you can do that manually:
image = np.frombuffer(stream_bytes, dtype="B").reshape(320, 240, 3)
… but if not, you have to send that information as part of your framing protocol as well.
Alternatively, you could send a pickle.dumps of the buffer and pickle.loads it on the other side, or np.save to a BytesIO and np.load the result. Either way, that includes the dtype, shape, order, and stride information as well as the raw bytes, so you don't have to worry about it.
The next problem is that you're exiting as soon as you display one image. Is that really what you want? If not… just don't do that.
But that just raises another problem. Do you really want to block the whole server with that cv.waitKey? Your client is capturing images and sending them as fast as it can; surely you either want to make the server display them as soon as they arrive, or change the design so the client only sends frames on demand. Otherwise, you're just going to get a bunch of near-identical frames, then a many-seconds-long gap while the client is blocked waiting for you to drain the buffer, then repeat.

How to download image from HTTP server [Python/sockets]

I want to download an example image from a HTTP server using methods defined in HTTP protocol (and socket's, of course).
I tried to implement it, but it seems that my code does not download the whole image, no matter if I have the while loop or not.
An example image is here: https://httpbin.org/image/png.
My code downloads only part of the image, and I do not know how to fix it. I do not want use any libraries, such as urllib, I want to use just the sockets.
Any ideas?
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('httpbin.org', 80))
s.sendall('GET /image/png HTTP/1.1\r\nHOST: httpbin.org\r\n\r\n')
reply = ""
while True:
data = s.recv(2048)
if not data: break
reply += data
# get image size
size = -1
tmp = reply.split('\r\n')
for line in tmp:
if "Content-Length:" in line:
size = int(line.split()[1])
break
headers = reply.split('\r\n\r\n')[0]
image = reply.split('\r\n\r\n')[1]
# save image
f = open('image.png', 'wb')
f.write(image)
f.close()

You are doing a HTTP/1.1 request. This HTTP version implicitly behaves like Connection: keep-alive was set. This means that the server might not close the TCP connection immediately after sending the response as you expect in your code but might keep the connection open to wait for more HTTP requests.
When replacing the version with HTTP/1.0 instead the server closes the connection after the request is done and the image is complete because HTTP/1.0 implies Connection: close.
Apart from that: HTTP is way more complex than you might think. Please don't just design your code after some example messages you've seen somewhere but actually read and follow the standards if you really want to implement HTTP yourself.

import socket
import select
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('httpbin.org', 80))
s.sendall(b'GET /image/png HTTP/1.1\r\nHOST: httpbin.org\r\n\r\n')
reply = b''
while select.select([s], [], [], 3)[0]:
data = s.recv(2048)
if not data: break
reply += data
headers = reply.split(b'\r\n\r\n')[0]
image = reply[len(headers)+4:]
# save image
f = open('image.png', 'wb')
f.write(image)
f.close()
Note this example is not perfect. The elegant way should be checking Content-Length header and recv exact length of data. (Instead of hard coding 3 seconds as timeout.) And if the server can use chunked encoding, it becomes even more complicated.)
--
The example is in python 3

Python socket send doesn't send the whole string

I am having trouble when I try to send a string from client to server over LAN
code for client:
def p():
os.system('tasklist > p.txt')
f = open('p.txt', 'r+')
proc = '''
'''
for line in f:
proc+=(line+'\b')
c.send(proc)
code for server:
def main():
while True:
command = raw_input('COMMAND >> ')
s.sendall(command)
data = s.recv(4096)
print(data+'\n\n')
and I am recieving only part of the data I sent.
I've already tried setting more bytes on s.recv, but that didn't really help, the problem persists.
What seems to be the problem?

Ref the docs https://docs.python.org/2/library/socket.html#socket.socket.send "Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data."
Check the return value from c.send(proc) for the total bytes sent and send more if needed.
E.g. if you are sending 100 chars, and c.send(data) returns 40, then you need to call c.send(data[40:]), and so on.
Alternatively, you might be able to/want to use sendall: https://docs.python.org/2/library/socket.html#socket.socket.sendall

python socket file transfer

I'm trying to write transfer files or chunks of data over a socket. I feel as if I'm reinventing the wheel, but my searches for a simple solution have failed (everything I find is either too simple or too complex). The server would run on a phone running python 2.5.4. The intended application would be to sync music files between the phone and a host computer.
This is the guts of what I have, which appears to work. I send and receive 'ok' to break up streams.
Is sending 'ok' back and forth essentially as stop bits to break up streams of data a reasonable technique?
Is there a standard way to do this?
Running any sort of library server (ftp, http) on the phone is not a useful solution given the limits of the phone's memory and processing power.
server:
import socket
c = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
c.bind(('', 1234))
c.listen(1)
s,a = c.accept()
while True:
data = s.recv(1024)
cmd = data[:data.find('\n')]
if cmd == 'get':
x, file_name, x = data.split('\n', 2)
s.sendall('ok')
with open(file_name, 'rb') as f:
data = f.read()
s.sendall('%16d' % len(data))
s.sendall(data)
s.recv(2)
if cmd == 'end':
s.close()
c.close()
break
client:
import socket
s = socket.socket()
s.connect(('192.168.1.2', 1234))
def get_file(s, file_name):
cmd = 'get\n%s\n' % (file_name)
s.sendall(cmd)
r = s.recv(2)
size = int(s.recv(16))
recvd = ''
while size > len(recvd):
data = s.recv(1024)
if not data:
break
recvd += data
s.sendall('ok')
return recvd
print get_file(s, 'file1')
print get_file(s, 'file2')
s.sendall('end\n')

Is sending 'ok' back and forth essentially as stop bits to break up
streams of data a reasonable technique?
Most protocols use some terminator or another. Popular alternatives are '\r\n', '\r\n\r\n' or EOF (ctrl+d), but these are just arbitrarily chosen and no worse or better than your 'ok', as long as your client and server know how to handle it.

Your code looks good.
You don't actually need to send across the size of the file. You can use while True, as the check if not data: break will stop the loop.
while True:
data = s.recv(1024)
if not data: print " Done "; break
recvd += data
Also, why are you sending 'ok' is the other side doesn't check for it? You are just skipping 2 bytes at each side.
Don't you need to cater to multiple clients? No need for multi-threading?

Is there a standard way to do this?
Yes. http://www.faqs.org/rfcs/rfc959.html
Describes the standard way to do this.
Here is an implementation: http://docs.python.org/library/ftplib.html

U may look at this implementation. It also take care of if the file is in a sub-directory. Here is the link!
server
import socket
import os
print('Waiting for clinet to connect...')
c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.bind(('', 1234))
c.listen(1)
s, a = c.accept()
print('Connected. Going to receive file.')
s.sendall('getfilename')
filename = s.recv(1024)
if '/' in filename:
dir = os.path.dirname(filename)
try:
os.stat(dir)
except:
print('Directory does not exist. Creating directory.')
os.mkdir(dir)
f = open(filename, 'wb')
print('Filename: ' + filename)
while True:
s.sendall('getfile')
size = int(s.recv(16))
print('Total size: ' + str(size))
recvd = ''
while size > len(recvd):
data = s.recv(1024)
if not data:
break
recvd += data
f.write(data)
#print(len(recvd))
break
s.sendall('end')
print('File received.')
s.close()
c.close()
f.close()
client
import socket
import sys
if len(sys.argv) > 1 :
print('Trying to connect...')
s = socket.socket()
s.connect(('127.0.0.1', 1234))
print('Connected. Wating for command.')
while True:
cmd = s.recv(32)
if cmd == 'getfilename':
print('"getfilename" command received.')
s.sendall(sys.argv[1])
if cmd == 'getfile':
print('"getfile" command received. Going to send file.')
with open(sys.argv[1], 'rb') as f:
data = f.read()
s.sendall('%16d' % len(data))
s.sendall(data)
print('File transmission done.')
if cmd == 'end':
print('"end" command received. Teminate.')
break

rsync is the standard way to sync files between two computers. You could write it in Python like this http://code.activestate.com/recipes/577518-rsync-algorithm/ or you could wrap the C library like this http://freshmeat.net/projects/pysync/ with some tweaks like replacing MD4 with MD5.
Or, if you want to do this at the socket level, you really should be using asynchat with asyncore. Here is an FTP server written with asynchat http://pyftpdlib.googlecode.com/svn-history/r20/trunk/pyftpdlib/FTPServer.py but you should start by reading http://www.doughellmann.com/PyMOTW/asynchat/ Pay attention to the part about Message Terminators point 2. A lot of network protocols do odd stuff like this, i.e. sometimes they send and receive full line commands and responses, and sometimes they send and receive chunks of arbitrary data preceded by the count of how many bytes are in the chunk. You can handle this much more easily with asynchat, and your program will scale much better too.