We Keep Coding

Is there a way to separate and clone an image area with cv2?

I know the title isn't very clear but its kinda a complicated idea and I don't have the slightest clue how else to explain it. If you can think of a better title please, please, please , tell me and ill edit the title.
My problem is this: One of my close coder pals challenged me to re-create this Image to Image translation example. There are a few catches though.
I cant use any pre-trained neural networks
I have to make it run realtime (use webcam)
So far I have made it up to having the face swapped but i need to make it go back on to the non webcam image but there is an issue. to do that I have to re-build and warp the whole image around that image including filling the background behind the face that isn't included in the the original source image. I have tried to use impainting but on some occasions it takes part of the hair and neck and merges it into the background just creating a mess of skin and hair colors. I have also tried expanding my mask on the cv2 impainting function but that results in a big background color square that also looks terrible. I assume the best solution would be to segment off the biggest area around the head with some sort of segmenting algorithm, then clone part of that area to preserve the background texture instead of creating a new bad texture and placing that cloned area inside the mask. This all has to be done in realtime.
In the end I need to copy the person in img1, then remove them (part im stuck on). then take the facial landmarks from img2 and map those landmarks onto img1's clone's face, then add the re-mapped img1 back onto the img1 background. I know i cant communicate these ideas as clearly as someone more qualified as i am in 8th grade so if any clarification is required please ask.

How do you move the mouse to all of the blue pixels on the screen's x y values averaged

I am trying to make a DIY Virtual Reality kit, and am not sufficient in Python. But by far, I think Python is the only way to achieve the following; The program takes each frame from the webcam feed, get's the blue pixels, and averages all of the XY positions of the pixels, and makes the mouse move there. In OpenCV and Python.
I've done a milion Google searches and cannot find what I need. I spent around 11 hours searching the OpenCV documentation and found nothing. I just need this program done. I'm running Python 3.7.3, and with OpenCV 2.
Any help would be much appreciated. I am fine with people giving me the full code as long as it is understandable and is legible.
Thank you.
*Edit, I am using Windows x86.

Convert your image to HSV. HSV is better for detection specific coloured areas generally. This SO link deals with the same issue as you described and have useful links on them. After you detect blue pixels with inRange function, use the mask to get an average of coordinate values(np.argwhere is useful for getting coordinates, then use np.mean over axis=0).

Aruco marker detection with 360 camera?

recently I have been playing with the 360 fly HD camera and wondering if Aruco Marker can be detected during real time. The first thing come to my mind is to convert the fisheye image into perspective image first and then perform the detection on the perspective image(I am gonna try it and will update my result here later).
Converting a fisheye image into a panoramic, spherical or perspective projection
Hugin HowTo: Convert 360 Image to Cropped Flat Panoramic Image
I am not an expert in this field. Has anyone done this before? Is this something can be achieved by calibrating the camera differently such as correcting the camera matrix and distortion coefficient matrix?
If I am heading to the wrong direction, please let me know.

I was able to get a better understanding during the process.
First, I want to say that 360(fisheye, spherical, however you call it) image is NOT distorted. I was so tricked by my intuition and thought that the image was distorted based on what it looks like. NO it is not distorted. Please read enter link description here for more information.
Next, I have tried both 360 fly cameras and neither works. Every time I tried to access the camera with opencv, it automatically powers off and switch to storage mode. I guess the 360 dev team purposely implements this switching function to prevent "hacking" of their products. But, I've seen people successfully hacked the 360 fly, it's definitely workable.
At last, I was able to detect Aruco with Ricoh theta V(theta S should also work). It's so developer friendly and I was able to make it run in my first attempt. You just have to select the right camera and let the code run. The only problem is the range, which is expected(about 6ft) and Ricoh camera is kind of expensive($499).
click here to view succesful detection

How to calculate Depth Information from USB 3.0 Stereo Camera?

I'm interested in using a stereo camera for calculating depth in video/images. The camera is a USB 3.0 Stereoscopic camera from Leopard Imaging https://www.leopardimaging.com/LI-USB30-V024STEREO.html. I'm using MAC OS X btw.
I was told by their customer support that it's a "UVC" camera. When connected to an apple computer it gives a greenish image.
My end goal is to use OpenCV to grab the left and right frames from both lenses so that I can calculate depth. I'm familiar with OpenCV, but not familiar with working with stereo cameras. Any help would be much appreciated. So far I have been doing this in Python 3:
import numpy as np
import cv2
import sys
from matplotlib import pyplot as plt
import pdb; pdb.set_trace()
print("Camera 1 capture", file=sys.stderr)
cap = cv2.VideoCapture(1)
print("Entering while", file=sys.stderr)
while(True):
_ = cap.grab()
retVal, frame = cap.retrieve()
#gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.imshow('frame', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
This works, but it only gives me a green picture/image with no depth. Any advice on how to get both left and right frames from the camera?

The leopard imaging people sent me a clue but I'm not able to make progress because i guess some missing file in my core file. However, I thought it might help somebody. So I'm posting it as an answer.
The message was sent by one of the leopard imaging people I contacted through mail. It goes like this.....
We have a customer who successfully separated the two videos on Linux OS one year ago. I tried to contact him to see if he can share the source code with us. Unfortunately, he already left the former company, but he still found some notes (below). Hope it helps.
The camera combines the image from the two sensors into one 16-bit pixel data (The high 8 bits from one camera, and the low 8 bit from the other camera).
To fix this problem in linux opencv, should skip color transform by opencv: at
modules/videoio/src/cap_v4l.cpp static IplImage* icvRetrieveFrameCAM_V4L( CvCaptureCAM_V4L* capture, int)
case V4L2_PIX_FMT_YUYV:
#if 1
/*
skip color convert. Just copy image buffer to frame.imageData
*/
memcpy(capture->frame.imageData, capture->buffers[capture->bufferIndex].start, capture->buffers[capture->bufferIndex].length);
#else
yuyv_to_rgb24(capture->form.fmt.pix.width, capture->form.fmt.pix.height,(unsigned char*) capture->buffers[capture->bufferIndex].start),
(unsigned char*)capture->frame.imageData);
#endif
Hope this helps.
DISCLAIMER : I went forward with the C++ version(my project was in C++) given in libuvc used the provided routines to obtain the left and right frames separately.

I am chasing the same issue, but with C/C++. I have contacted Leopard and am waiting for an answer. My understanding is that the two grayscale cameras are interlaced into a single image (and I think that OpenCV sees this as a color image, hence the strange colors and being out of focus.) You need to break apart the bytes into two separate frames. I am experimenting, trying to figure out the byte placement, but have not gotten too far. If you figure this out, please let me know!
They have a C# on Windows example here:
https://www.dropbox.com/sh/49cpwx0s70fuich/2e0_mFTJY_
Unfortunately it is using their libraries (which are not source) to do the heavy lifting, so I can't figure out what they are doing.

I met the same issue as you and finally arrived to a solution. But I don't know if OpenCV can handle it directly, especially in Python.
As jordanthompson said, the two images are interlaced into one. The image you receive is in YUYV (Y is the light intensity, UV contains the color information). Each pixel is coded on 16 bits, 8 for Y, 8 for U or V depending on which pixel you are looking at.
Here, the Y bits come from the left image, the UV bits from the right image. But when OpenCV receives this image, it converts it to RGB which then definitely mix the two images.. I could not find a way in Python to tell OpenCV to get the image without converting it... Therefore we need to read the image before OpenCV. I managed to do it with libuvc (https://github.com/ktossell/libuvc) after adding two small functions to perform the proper conversions. I guess you can use this library if you can use openCV in C++ instead of Python. If you really have to stick with Python, then I don't have a complete solution, but at least now you know what to look for: try to read the image directly in YUYV and then separate the bits into left and right image (you will get two grayscale images).
Good luck !

How to scale an image without occasionally inverting it (with the Python Imaging Library)

When resizing images along the lines shown in this question occasionally the resulting image is inverted. About 1% of the images I resize are inverted, the rest is fine. So far I was unable to find out what is different about these images.
See resized example and original image for examples.
Any suggestions on how to track down that problem?

I was finally able to find someone experienced in JPEG and with some additional knowledge was able to find a solution.
JPEG is a very underspecified
Format.
The second image is a valid JPEG but it is in CMYK color space, not in RGB color space.
Design minded tools (read: things from Apple) can process CMYK JPEGs, other stuff (Firefox, IE) can't.
CMYK JPEG is very under specified and the way Adobe Photoshop writes it to disk is borderline to buggy.
Best of it all there is a patch to fix the issue.

Your original image won't display for me; Firefox says
The image “http://images.hudora.de/o/NIRV2MRR3XJGR52JATL6BOVMQMFSV54I01.jpeg”
cannot be displayed, because it contains errors.
This suggests that the problem arises when you attempt to resize a corrupted JPEG, and indeed your resized example shows what looks like JPEG corruption to my eye (Ever cracked open a JPEG image and twiddled a few bits to see what it does to the output? I have, and a few of my abominable creations looked like that). There are a few JPEG repair tools out there, but I've never seriously tried any of them and don't know if they might be able to help you out.