I was practicing OpenCV on google colaboratory becasuse I don't know how to use OpenCV on GPU, when I run OpenCV on my hardware, It takes a lot of CPU, so I went to Google colaboratory.
The link to my notebook is here.
If you don't want to watch it, then here is the code:
import cv2
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
cap = cv2.VideoCapture(0)
while True:
_, img = cap.read()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.1, 4)
for (x, y, w, h) in faces:
cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2)
cv2.imshow('img', img)
k = cv2.waitKey(30) & 0xff
if k==27:
break
cap.release()
The same code worked fine on my PC, but not on Google Colaboratory. The error is:
---------------------------------------------------------------------------
error Traceback (most recent call last)
<ipython-input-5-0d9472926d8c> in <module>()
6 while True:
7 _, img = cap.read()
----> 8 gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
9 faces = face_cascade.detectMultiScale(gray, 1.1, 4)
10 for (x, y, w, h) in faces:
error: OpenCV(4.1.2) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
PS~I have the haarcascade file inside the same directory of my notebook in Google Colaboratory
How to deal with it? If not then is there any "concrete" solution to run OpenCV on my CUDA enabled GPU instead of CPU? Thanks in advance!
_src.empty() means that it had problem to get frame from camera and img is None and when it tries cvtColor(None, ...) then it gives _src.empty().
You should check if img is not None: because cv2 doesn't raise error when it can't get frame from camera or read image from file. And sometimes camera needs time to "warm up" and it can gives few empty frames (None).
VideoCapture(0) reads frame from camera directly connected to computer which runs this code - and when you run code on server Google Colaboratory then it means camera connected directly to server Google Colaboratory (not your local camera) but this server doesn't have camera so VideoCapture(0) can't work on Google Colaboratory.
cv2 can't get image from your local camera when it runs on server. Your web browser may have access to your camera but it needs JavaScript to get frame and send to server - but server needs code to get this frame
I checked in Google if Google Colaboratory can access local webcam and it seems they created script for this - Camera Capture - in first cell is function take_photo() which uses JavaScript to access your camera and display in browser, and in second cell this function is used to display image from local camera and to take screenshot.
You should use this function instead of VideoCapture(0) to work on server with your local camera.
BTW: Belove take_photo() there is also information about cv2.im_show() because it also works only with monitor directly connected to computer which runs this code (and this computer has to run GUI like Windows on Windows , X11 on Linux) - and when you run it on server then it want to display on monitor directly connected to server - but server usually works without monitor (and without GUI)
Google Colaboratory has special replacement which displays in web browser
from google.colab.patches import cv2_imshow
BTW: If you will have problem with loading haarcascades .xml then you may need folder to filename. cv2 has special variable for this cv2.data.haarcascades
path = os.path.join(cv2.data.haarcascades, 'haarcascade_frontalface_default.xml')
cv2.CascadeClassifier( path )
You can also see what is in this folder
import os
filenames = os.listdir(cv2.data.haarcascades)
filenames = sorted(filenames)
print('\n'.join(filenames))
EDIT:
I created code which can get from local webcam frame by frame without using button and without saving in file. Problem is that it is slow - because it still have to send frame from local web browser to google colab server and later back to local web browser
Python code with JavaScript functions
#
# based on: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=2viqYx97hPMi
#
from IPython.display import display, Javascript
from google.colab.output import eval_js
from base64 import b64decode, b64encode
import numpy as np
def init_camera():
"""Create objects and functions in HTML/JavaScript to access local web camera"""
js = Javascript('''
// global variables to use in both functions
var div = null;
var video = null; // <video> to display stream from local webcam
var stream = null; // stream from local webcam
var canvas = null; // <canvas> for single frame from <video> and convert frame to JPG
var img = null; // <img> to display JPG after processing with `cv2`
async function initCamera() {
// place for video (and eventually buttons)
div = document.createElement('div');
document.body.appendChild(div);
// <video> to display video
video = document.createElement('video');
video.style.display = 'block';
div.appendChild(video);
// get webcam stream and assing to <video>
stream = await navigator.mediaDevices.getUserMedia({video: true});
video.srcObject = stream;
// start playing stream from webcam in <video>
await video.play();
// Resize the output to fit the video element.
google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);
// <canvas> for frame from <video>
canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
//div.appendChild(input_canvas); // there is no need to display to get image (but you can display it for test)
// <img> for image after processing with `cv2`
img = document.createElement('img');
img.width = video.videoWidth;
img.height = video.videoHeight;
div.appendChild(img);
}
async function takeImage(quality) {
// draw frame from <video> on <canvas>
canvas.getContext('2d').drawImage(video, 0, 0);
// stop webcam stream
//stream.getVideoTracks()[0].stop();
// get data from <canvas> as JPG image decoded base64 and with header "data:image/jpg;base64,"
return canvas.toDataURL('image/jpeg', quality);
//return canvas.toDataURL('image/png', quality);
}
async function showImage(image) {
// it needs string "data:image/jpg;base64,JPG-DATA-ENCODED-BASE64"
// it will replace previous image in `<img src="">`
img.src = image;
// TODO: create <img> if doesn't exists,
// TODO: use `id` to use different `<img>` for different image - like `name` in `cv2.imshow(name, image)`
}
''')
display(js)
eval_js('initCamera()')
def take_frame(quality=0.8):
"""Get frame from web camera"""
data = eval_js('takeImage({})'.format(quality)) # run JavaScript code to get image (JPG as string base64) from <canvas>
header, data = data.split(',') # split header ("data:image/jpg;base64,") and base64 data (JPG)
data = b64decode(data) # decode base64
data = np.frombuffer(data, dtype=np.uint8) # create numpy array with JPG data
img = cv2.imdecode(data, cv2.IMREAD_UNCHANGED) # uncompress JPG data to array of pixels
return img
def show_frame(img, quality=0.8):
"""Put frame as <img src="data:image/jpg;base64,...."> """
ret, data = cv2.imencode('.jpg', img) # compress array of pixels to JPG data
data = b64encode(data) # encode base64
data = data.decode() # convert bytes to string
data = 'data:image/jpg;base64,' + data # join header ("data:image/jpg;base64,") and base64 data (JPG)
eval_js('showImage("{}")'.format(data)) # run JavaScript code to put image (JPG as string base64) in <img>
# argument in `showImage` needs `" "`
And code which uses it in loop
#
# based on: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=zo9YYDL4SYZr
#
#from google.colab.patches import cv2_imshow # I don't use it but own function `show_frame()`
import cv2
import os
face_cascade = cv2.CascadeClassifier(os.path.join(cv2.data.haarcascades, 'haarcascade_frontalface_default.xml'))
# init JavaScript code
init_camera()
while True:
try:
img = take_frame()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
#cv2_imshow(gray) # it creates new image for every frame (it doesn't replace previous image) so it is useless
#show_frame(gray) # it replace previous image
faces = face_cascade.detectMultiScale(gray, 1.1, 4)
for (x, y, w, h) in faces:
cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2)
#cv2_imshow(img) # it creates new image for every frame (it doesn't replace previous image) so it is useless
show_frame(img) # it replace previous image
except Exception as err:
print('Exception:', err)
I don't use from google.colab.patches import cv2_imshow because it always add new image on page instead of replacing existing image.
The same code as Notebook on Google Colab:
https://colab.research.google.com/drive/1j7HTapCLx7BQUBp3USiQPZkA0zBKgLM0?usp=sharing
The possible problem in the code is, you need to give full-path as the directory when using Haar-like features.
face_cascade = cv2.CascadeClassifier('/User/path/to/opencv/data/haarcascades/haarcascade_frontalface_default.xml')
The colab issue with opencv has been known for quite some time, also the same question asked here
As stated here, you can use the cv2_imshow to display the image, but you want to process Camera frames.
from google.colab.patches import cv2_imshow
img = cv2.imread('logo.png', cv2.IMREAD_UNCHANGED)
cv2_imshow(img)
One possible answer:
Insert Camera Capture snippet, the method take_photobut you need to modify the method.
face_cascade = cv2.CascadeClassifier('/opencv/data/haarcascades/haarcascade_frontalface_default.xml')
try:
filename = take_photo()
img = Image(filename)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.1, 4)
for (x, y, w, h) in faces:
cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2)
cv2_imshow("img", img)
except Exception as err:
print(str(err))
The above code requires editing since there is no direct way to use VideoCapture you have to modify take_photo
Related
I'm trying to take a screenshot with python. But every option I try only seems to return the desktop wallpaper and not the programs on top.
Here's a run through of what I've tried and how I'm doing it.
First I used autopy to try and get pixels from the screen using autopy.color.hex_to_rgb(autopy.screen.get_color(x, y)). But it was only telling me the pixels of the desktop background.
Then I tried PIL(Pillow) using this code:
from PIL import ImageGrab
im = ImageGrab.grab()
im.save('screenshot.png')
This only returned the desktop wallpaper.
Finally, I've tried using this script which I found on this thread.
import Quartz
import LaunchServices
from Cocoa import NSURL
import Quartz.CoreGraphics as CG
def screenshot(path, region = None):
"""region should be a CGRect, something like:
>>> import Quartz.CoreGraphics as CG
>>> region = CG.CGRectMake(0, 0, 100, 100)
>>> sp = ScreenPixel()
>>> sp.capture(region=region)
The default region is CG.CGRectInfinite (captures the full screen)
"""
if region is None:
region = CG.CGRectInfinite
# Create screenshot as CGImage
image = CG.CGWindowListCreateImage(
region,
CG.kCGWindowListOptionOnScreenOnly,
CG.kCGNullWindowID,
CG.kCGWindowImageDefault)
dpi = 72 # FIXME: Should query this from somewhere, e.g for retina displays
url = NSURL.fileURLWithPath_(path)
dest = Quartz.CGImageDestinationCreateWithURL(
url,
LaunchServices.kUTTypePNG, # file type
1, # 1 image in file
None
)
properties = {
Quartz.kCGImagePropertyDPIWidth: dpi,
Quartz.kCGImagePropertyDPIHeight: dpi,
}
# Add the image to the destination, characterizing the image with
# the properties dictionary.
Quartz.CGImageDestinationAddImage(dest, image, properties)
# When all the images (only 1 in this example) are added to the destination,
# finalize the CGImageDestination object.
Quartz.CGImageDestinationFinalize(dest)
if __name__ == '__main__':
# Capture full screen
screenshot("/tmp/testscreenshot_full.png")
# Capture region (100x100 box from top-left)
region = CG.CGRectMake(0, 0, 100, 100)
screenshot("/tmp/testscreenshot_partial.png", region=region)
Again it returns my desktop wallpaper.
Why is it doing this? How can I get a screenshot in the same way I would if I were to press 'cmd + shift + 3'.
This is a privacy feature, macOS prevents arbitrary apps from viewing the contents of other app windows. To get permission, your app will need access to the screen recording permission in macOS preferences.
Since this is a Python script, I think the app you're running the script from needs to be given permission, so either Terminal or whatever IDE you're using.
I have a question on threading in python.
I have 3 applications for which I want to access the same resource (a camera). I would like to start the camera up streaming frames and then each of these processes take frames as needed and at different resolutions (or I can scale the images later if that is not efficient). I can get each of these three applications to run independently but have not found a way to thread them successfully. All three have been learned from others and I am grateful for them sharing. Credits below.
Application 1 streams the video feed to a web server.
with picamera.PiCamera(resolution='640x480', framerate=16) as camera:
output = StreamingOutput()
camera.start_recording(output, format='mjpeg')
try:
address = ('', 8000)
server = StreamingServer(address, StreamingHandler)
server.serve_forever()
finally:
camera.stop_recording()
Application 2 grabs small frames to memory often for detecting motion and then if certain conditions are met an image is saved to disk. In this case I want to grab low resolution frames for speeding up the analysis and also keeping them in memory only. For the part of saving I am interested in grabbing a higher resolution file for disk
def captureTestImage(settings, width, height):
command = "raspistill %s -w %s -h %s -t 200 -e bmp -n -o -" % (settings, width, height)
imageData = io.BytesIO()
imageData.write(subprocess.check_output(command, shell=True))
imageData.seek(0)
im = Image.open(imageData)
buffer = im.load()
imageData.close()
return im, buffer
while True:
motionState = P3picam.motion()
if motionState:
with picamera.PiCamera() as camera:
camera.resolution = (1920,1080)
camera.capture(picPath + picName)
Application 3 grabs frames from a video stream and then uses openCV on them.
vs = VideoStream(src=0).start()
time.sleep(2.0)
while True:
frame = vs.read()
frame = imutils.resize(frame, width=500)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
Application 1 credits: http://picamera.readthedocs.io/en/latest/recipes2.html#web-streaming
Application 2 credits: http://pastebin.com/raw.php?i=yH7JHz9w
Application 3 credits: https://www.pyimagesearch.com/2018/06/25/raspberry-pi-face-recognition/
I tried to use the following codes to play YouTube videos on Google Colab,
!pip install pytube==9.1.0
from pytube import YouTube
yt = YouTube('https://www.youtube.com/watch?v=-ncJV0tMAjE&t=72s')
The title of yt object is,
print(yt.title)
>> 作業2 第二題 強健控制：結構化不確定參數的Riccati equation
Then, I imported cv2.
import cv2
import numpy as np
cap = cv2.VideoCapture('作業2 第二題 強健控制：結構化不確定參數的Riccati equation.mp4')
while(cap.isOpened()):
ret, frame = cap.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.imshow('frame',gray)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
However, I always got an error message as,
Runtime died. Automatically restarting.
What should I do?
If you want to play a youtube video on google colab,
from IPython.display import YouTubeVideo
YouTubeVideo('-ncJV0tMAjE')
'-ncJV0tMAjE' is from the URL(https://www.youtube.com/watch?v=-ncJV0tMAjE) after 'watch?v='
You need to get the embed code of your video.
from IPython.display import HTML
HTML('<iframe width="560" height="315" src="https://www.youtube.com/embed/lTTZ8PkQ_Pk" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')
That is because cv2.imshow destroys your session in google colab, instead of that you have to import cv2_imshow: from google.colab.patches import cv2_imshow, but cv2_imshow just takes one argument and that is your frame or image.
I'm using Basler camera and python to record some video. I can successfully capture individual frames, but I don't know how to record a video.
Following is my code:
import os
import pypylon
from imageio import imwrite
import time
start=time.time()
print('Sampling rate (Hz):')
fsamp = input()
fsamp = float(fsamp)
time_exposure = 1000000*(1/fsamp)
available_cameras = pypylon.factory.find_devices()
cam = pypylon.factory.create_device(available_cameras[0])
cam.open()
#cam.properties['AcquisitionFrameRateEnable'] = True
#cam.properties['AcquisitionFrameRate'] = 1000
cam.properties['ExposureTime'] = time_exposure
buffer = tuple(cam.grab_images(2000))
for count, image in enumerate(buffer):
filename = str('I:/Example/{}.png'.format(count))
imwrite(filename, image)
del buffer
I haven't found a way to record a video using pypylon; it seems to be a pretty light wrapper around Pylon. However, I have found a way to save a video using imageio:
from imageio import get_writer
with get_writer('I:/output-filename.mp4', fps=fps) as writer:
# Some stuff with the frames
The above can be used with .mov, .avi, .mpg, .mpeg, .mp4, .mkv or .wmv, so long as the FFmpeg program is available. How you will install this program depends on your operating system. See this link for details on the parameters you can use.
Then, simply replace the call to imwrite with:
writer.append_data(image)
ensuring that this occurs in the with block.
An example implementation:
import os
import pypylon
from imageio import get_writer
while True:
try:
fsamp = float(input('Sampling rate (Hz): '))
break
except ValueError:
print('Invalid input.')
time_exposure = 1000000 / fsamp
available_cameras = pypylon.factory.find_devices()
cam = pypylon.factory.create_device(available_cameras[0])
cam.open()
cam.properties['ExposureTime'] = time_exposure
buffer = tuple(cam.grab_images(2000))
with get_writer(
'I:/output-filename.mkv', # mkv players often support H.264
fps=fsamp, # FPS is in units Hz; should be real-time.
codec='libx264', # When used properly, this is basically
# "PNG for video" (i.e. lossless)
quality=None, # disables variable compression
pixelformat='rgb24', # keep it as RGB colours
ffmpeg_params=[ # compatibility with older library versions
'-preset', # set to faster, veryfast, superfast, ultrafast
'fast', # for higher speed but worse compression
'-crf', # quality; set to 0 for lossless, but keep in mind
'11' # that the camera probably adds static anyway
]
) as writer:
for image in buffer:
writer.append_data(image)
del buffer
I am trying to implement a Flask backend endpoint where two images can be uploaded, a background and a foreground (the foreground has a transparent background), and it will paste the foreground on top of the background. I have the following Python code that I have tested with local files and works:
background_name = request.args.get("background")
overlay_name = request.args.get("overlay")
output_name = request.args.get("output")
background_data = request.files["background"].read()
overlay_data = request.files["overlay"].read()
background = Image.open(BytesIO(background_data))
overlay = Image.open(BytesIO(overlay_data))
background = background.convert("RGBA")
overlay = overlay.convert("RGBA")
overlay = resize_image(overlay, background.size)
background.paste(overlay, (0, 0, background.size[0], background.size[1],), overlay)
background.save(output_name, "PNG")
However, when I try to upload the same images through Alamofire and the following code:
Alamofire.upload(multipartFormData: { formData in
formData.append(bgData, withName: "background", fileName: "background.jpg", mimeType: "image/jpeg")
formData.append(fgData, withName: "foreground", fileName: "foreground.png", mimeType: "image/png")
}, to: "http://localhost:8080/image_overlay?background=background%2Ejpg&overlay=overlay%2Epng&output=result%2Epng", encodingCompletion: { result in
switch result {
case .success(let upload, _, _):
upload.validate().responseJSON(completionHandler: { response in
switch response.result {
case .success(let value): print("success: \(value)")
case .failure((let error)): print("response error \(error)")
}
})
case .failure(let error):
print("encoding error \(error)")
}
})
The foreground appears with a white background instead of a transparent background and the resulting image is just the foreground with a white background. How can I get Alamofire to send the transparency?
EDIT:
I tried translating this to a cURL request and it works as expected. I used nc-l localhost 8080 to view the full request, and it seems that with the foreground picture, even though the Content-Type was set to "application/octet-stream", the next line was "?PNG". This line was no present from the Alamofire request. How can I get the request to recognize the image as a PNG?
I can't believe I was doing this, but when I was obtaining to image data, I was using
let fgData = UIImageJPEGRepresentation(UIImage(named: "overlay.png"), 0.75)
This was the only way I had ever obtained data from an image in Swift, even though obviously it should be what I changed it to:
let fgData = UIImagePNGRepresentation(UIImage(named: "overlay.png"))