We Keep Coding

Sending Real Time images captured using unity camera

server
private void SendImageByte()
{
image_bytes = cm.Capture();
print(image_bytes.Length);
if (connectedTcpClient == null)
{
return;
}
try
{
// Get a stream object for writing.
NetworkStream stream = connectedTcpClient.GetStream();
if (stream.CanWrite)
{
// string serverMessage = "This is a message from your server.";
// Convert string message to byte array.
byte[] serverMessageAsByteArray = Encoding.ASCII.GetBytes(image_bytes.ToString());
// Write byte array to socketConnection stream.
stream.Write(serverMessageAsByteArray, 0, serverMessageAsByteArray.Length);
Debug.Log("Server sent his message - should be received by client");
}
}
catch (SocketException socketException)
{
Debug.Log("Socket exception: " + socketException);
}
}
client
import socket
host = "127.0.0.1"
port = 1755
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
def receive_image():
data = sock.recv(999999).decode('utf-8')
print(len(data))
while True:
receive_image()
here script capture images from unity camera
public byte[] Capture()
{
if(renderTexture == null)
{
// creates off-screen render texture that can rendered into
rect = new Rect(0, 0, captureWidth, captureHeight);
renderTexture = new RenderTexture(captureWidth, captureHeight, 24);
screenShot = new Texture2D(captureWidth, captureHeight, TextureFormat.RGB24, false);
}
// _camera = GetComponent<Camera>();
_camera.targetTexture = renderTexture;
_camera.Render();
// reset active camera texture and render texture
_camera.targetTexture = null;
RenderTexture.active = null;
// read pixels will read from the currently active render texture so make our offscreen
// render texture active and then read the pixels
RenderTexture.active = renderTexture;
screenShot.ReadPixels(rect, 0, 0);
screenShot.Apply();
byte[] imageBytes = screenShot.EncodeToPNG();
//Object.Destroy(screenShot);
//File.WriteAllBytes(Application.dataPath + "/../"+ imagePath + "/img{counter}.png", bytes);
//counter = counter + 1;
return imageBytes;
}
Am trying to send real-time images on Unity3D from C# to python using socket communication to be processed and return back values to unity, but the problem even the bytes length received on the client is not the same as the server. I send about 400K bytes but I receive only 13
C# is the server and python is the client
or am doing it wrong but the main goal I want to create simulator work as udacity self-driving

Are you sure that image_bytes.ToString() returns what you expect and not maybe just something like "System.Byte[]" => 13 chars => 13 bytes.
In general why would you convert an already byte[] into a string just in order to convert it back into byte[] to send? I'm pretty sure you do not want to transmit binary image data using UTF-8 ... one option might be Base64 string, but still that would be quite inefficient.
Just send the raw bytes like e.g.
stream.Write(image_bytes, 0, image_bytes.Length);
And then receive until you receive that length.
A typical solution is to prepend the length of the message to send and on the receiver side actually wait until you received exactly that amount of bytes like e.g.
var lengthBytes = BitConverter.GetBytes(image_bytes.Length);
stream.Write(lengthBytes, 0, lengthBytes.Length);
stream.Write(image_bytes, 0, image_bytes.Length);
Now you know that on the receiver side you first have to receive exactly 4 bytes (== one int) which will tell you exactly how many bytes to receive for the actual payload.
Now I'm no python expert but after googling around a bit I think something like
def receive_image()
lengthBytes = sock.recv(4)
length = struct.unpack("!i", lengthBytes)[0]
data = sock.recv(length)
Note: After reading John Gordon's comment on the question I guess this still doesn't fully solve the waiting until according buffers are actually filled - as said no python expert - but I hope it gives you a idea where to go ;)

How to send an int from Python to an Arduino in order to use it as an argument for the neopixel function setPixelcolor()?

I am trying to send an int number from Python to an Arduino using PySerial, using .write([data]) to send with Python and Serial.read() or Serial.readString() to recieve on the Arduino, then .setPixelColor() and .show() to light a LED on a matrix which position corresponds to the int sent by the arduino (I am using the Duinofun Neopixel Shield).
But It does not seem to work properly, and I can't use the Serial Monitor as I am sending my data as the port would be busy.
I have tried to input a number myself using Serial.readString() then converting the string to an int and finally putting in into my function that displays the LED.
It does work properly when I do this, but when I send some data over, all the previously lit LEDs suddenly switch off which can only be caused by a reset of the Arduino board as far as I know.
This is the python code, it simply sends an int chosen by the user
import serial
a = int(input('Enter pixel position : '))
ser = serial.Serial("COM16", 9600)
ser.write([a])
And this is the part of the Arduino program that reads the incoming data.
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(40, 6, NEO_GRB + NEO_KHZ800);
void setup() {
// put your setup code here, to run once:
pixels.begin();
Serial.begin(9600);
}
void loop() {
String a = Serial.readString();
int b = a.toInt();
pixels.setPixelColor(b, 30,30,30);
pixels.show();
Serial.println(a);
delay(1000);
}
All the LED switch off when I send some data, except the first LED which position corresponds to a 0 used in the .setPixelColor() function.
Problem is, the LED should light to the corresponding int sent by Python (e.g light the fifth LED for an int of 4).

You don't need to send an int from your Python script. Just send a string and then convert it back to int on your Arduino. Also, you can verify the number simply on your Arduino code if the received value is valid.
Another problem with your Arduino code is you are not checking the Serial port availability which would return an empty string by Serial.readString().
A simple approach is shown below but you can extend it for other pixels.
Python script:
import serial
ser = serial.Serial("COM16", 9600)
while True:
input_value = input('Enter pixel position: ')
ser.write(input_value.encode())
Arduino code:
#define MIN_PIXEL_RANGE 0
#define MAX_PIXEL_RANGE 100
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(40, 6, NEO_GRB + NEO_KHZ800);
void setup()
{
// put your setup code here, to run once:
pixels.begin();
Serial.begin(9600);
}
void loop()
{
if (Serial.available())
{
String a = Serial.readString();
Serial.print("Received Value: ");
Serial.println(a);
int b = a.toInt();
if ((b >= MIN_PIXEL_RANGE) && (b <= MAX_PIXEL_RANGE))
{
pixels.setPixelColor(b, 30, 30, 30);
pixels.show();
delay(1000);
}
}
}

You can communicate between Ardinos and Python really easily and reliably if you use the pip-installable package pySerialTransfer. The package is non-blocking, easy to use, supports variable length packets, automatically parses packets, and uses CRC-8 for packet corruption detection.
Here's an example Python script:
from pySerialTransfer import pySerialTransfer as txfer
if __name__ == '__main__':
try:
link = txfer.SerialTransfer('COM13')
link.txBuff[0] = 'h'
link.txBuff[1] = 'i'
link.txBuff[2] = '\n'
link.send(3)
while not link.available():
if link.status < 0:
print('ERROR: {}'.format(link.status))
print('Response received:')
response = ''
for index in range(link.bytesRead):
response += chr(link.rxBuff[index])
print(response)
link.close()
except KeyboardInterrupt:
link.close()
Note that the Arduino will need to use the library SerialTransfer.h. You can install SerialTransfer.h using the Arduino IDE's Libraries Manager.
Here's an example Arduino sketch:
#include "SerialTransfer.h"
SerialTransfer myTransfer;
void setup()
{
Serial.begin(115200);
Serial1.begin(115200);
myTransfer.begin(Serial1);
}
void loop()
{
myTransfer.txBuff[0] = 'h';
myTransfer.txBuff[1] = 'i';
myTransfer.txBuff[2] = '\n';
myTransfer.sendData(3);
delay(100);
if(myTransfer.available())
{
Serial.println("New Data");
for(byte i = 0; i < myTransfer.bytesRead; i++)
Serial.write(myTransfer.rxBuff[i]);
Serial.println();
}
else if(myTransfer.status < 0)
{
Serial.print("ERROR: ");
Serial.println(myTransfer.status);
}
}
Lastly, note that you can transmit ints, floats, chars, etc. using the combination of these libraries!

Getting IOError: [Errno 121] Remote I/O error with smbus on python (raspberry) while trying to get data over I2C from Arduino

I am facing problems, that pyhton throws me on my raspberry pi 3 sometimes this IOError during starting a script which is requesting data from an Arduino over I2C.
Electrical connection is perfect so this is not the issues.
Furthermore I also dont get any errors while using i2cget -y 1 0x04
Only the python scripts sucks sometime and I dont know why.
This is my Arduino Code:
I register an onReceive and an onRequestEvent.
onReceive Callback will define what kind of data should be send back to the raspberry.
onRequest Callback does the response.
#include <CommonFunction.h>
#include <Wire.h>
#define I2C_ADDRESS 0x4
commonFunc GetCountsEverySecond;
int g_iOnRequestActionCode = 0;
unsigned long g_lSecondsSinceStart = 0;
void setup()
{
Wire.begin(I2C_ADDRESS);
Wire.onRequest(sendDataOverI2CGateway);
Wire.onReceive(defineOnRequestAction);
}
void loop()
{
tickSeconds();
}
void tickSeconds()
{
if (GetCountsEverySecond.TimeTriggerAt(1000))
{
g_lSecondsSinceStart++;
}
}
void sendOperationTimeDataOverI2C()
{
unsigned long longInt = g_lSecondsSinceStart;
byte size = sizeof(longInt);
byte arr[size];
for (int i = 0; i < size; i++)
{
int iBitShift = 8 * (size - i - 1);
if (iBitShift >= 8)
arr[i] = ((longInt >> iBitShift) & 0xFF);
else
arr[i] = (longInt & 0xFF);
}
Wire.write(arr, size);
g_bI2CSending = true;
}
void sendDataOverI2CGateway()
{
switch(g_iOnRequestActionCode)
{
case 0:
sendRainDataOverI2C();
break;
case 1: // send firmware version
sendVersionDataOverI2C();
break;
case 2: // send operation time of arduino in seconds from start
sendOperationTimeDataOverI2C();
break;
default: break;
}
}
void defineOnRequestAction(int iBuffer)
{
while (Wire.available())
{
g_iOnRequestActionCode = Wire.read();
}
}
Here is my python Code.
Pretty straight forward but it causes some headache.
import smbus
import time
bus = smbus.SMBus(1)
while True:
data = bus.read_i2c_block_data(0x04,0x02,4)
result = 0
for b in data:
result = result * 256 + int(b)
print(result)
time.sleep(1)
After executing my python script I am getting sometime this error:
pi#WeatherStation:~/workspace $ sudo python readTimeOperationData.py
Traceback (most recent call last):
File "readTimeOperationData.py", line 5, in <module>
data = bus.read_i2c_block_data(0x04,0x02,4)
IOError: [Errno 121] Remote I/O error
Can anyone help me to fix this issue?
Cheers Dieter

I solved it!!
I got a hint from this post:
https://www.raspberrypi.org/forums/viewtopic.php?t=203286
By adding a delay after bus = smbus.SMBus(1) solved this issue.
It seems that a short delay is somehow needed so that the I2C can settle.
Working Code tested by calling script 100times without issues.
import smbus
import time
bus = smbus.SMBus(1)
time.sleep(1) #wait here to avoid 121 IO Error
while True:
data = bus.read_i2c_block_data(0x04,0x02,4)
result = 0
for b in data:
result = result * 256 + int(b)
print(result)
time.sleep(1)

This worked for me
import smbus
import time
bus = smbus.SMBus(1)
time.sleep(1)
if you use RPLCD library, add this code before initialize the object
bus = smbus.SMBus(1)
time.sleep(1) #wait here to avoid 121 IO Error
lcd = CharLCD('PCF8574', 0x27)
lcd.cursor_pos = (0,0)
thank you

Send a String from Python to Arduino LCD

I want to display a string on an Arduino LCD 16x2 using python, but I’ve encountered problems with serial communication.
Here is the code running in Arduino:
Arduino Code
#include <LiquidCrystal.h>
LiquidCrystal lcd(8, 9, 4, 5, 6, 7);
String stringa;
const unsigned long TimeOut = 10; // timeout 10 ms
String stringa1;
String stringa2;
void setup() {
lcd.begin(16, 2);
Serial.begin(9600);
}
void loop() {
stringa = "";
unsigned long T = 0; // timer
T = millis(); // timer running
while (millis() - T < TimeOut) {
// waiting timeout
while (Serial.available() > 0) {
// receiving Serial
stringa += char(Serial.read()); // add char
T = millis(); // reset timer
}
}
if (stringa.length() > 32) {
lcd.setCursor(0, 1);
lcd.print("stringa length: " + stringa.length());
delay(2000);
lcd.print(" ");
} else {
stringa1 = stringa.substring(0 , 16);
stringa2 = stringa.substring(16);
lcd.setCursor(0, 0);
lcd.print(stringa1);
lcd.setCursor(0, 1);
lcd.print(stringa2);
delay(5000);
}
}
It works perfectly with Serial communication from Keyboard provided in Arduino IDE. But it doesn't work when I try to send a string using the Python script below:
Python Code
import serial
import sys
import time
arduino = serial.Serial('COM3', 9600, timeout=0)
stringa = 'hello'
arduino.write(bytes(stringa,'utf-8'))
arduino.close()
Where is the problem? I can't find a solution! Thanks.

Take a look at the difference between the timeouts in the C file above and the python script below.
The timeout is 10 milliseconds in your C file wheareas it’s 0 in your Python script. Also check the result of the arduino.write() to make sure that it was successful.
Possibly implement something like the following:
import serial
import sys
import time
arduino = serial.Serial('COM3', 9600, timeout=10)
stringa = 'hello'
try:
arduino.write(stringa.encode())
except OsError:
print "Write failed!"
arduino.close()
If this does not work then try checking the serial ports between both the C file and the Python script. Make sure they are the same. Hope this helps!

XBee S2 Wireless Sensor Network

I'm working on a wireless sensor network in which I have one coordinator router (API mode 2) connected to a Raspberry Pi 2, 5 or more routers in API mode 2 as well. Each router is connected to an Arduino Uno. The Unos also have different sensors attached to them (temperature, humidity etc). I have to send data from the sensors to the coordinator and process it. I have successfully transferred data using one router and coordinator (just two XBee S2 modules). On the Arduini I'm using Andrew's library https://github.com/andrewrapp/xbee-arduino and on the Pi I'm using a Python-xbee library https://github.com/nioinnovation/python-xbee. For a single router and coordinator my codes are:
Arduino Code (Router):
#include <XBee.h>
#include <math.h>
// create the XBee object
XBee xbee = XBee();
int sensor = A5;
uint8_t payload[8] = {0, 0, 0, 0, 0, 0, 0, 0};
// union to convert float to byte string
union u_tag {
uint8_t b[4];
float fval;
} u;
// SH + SL Address of receiving XBee
XBeeAddress64 addr64 = XBeeAddress64(0x0013a200, 0x40DC7C90);
ZBTxRequest zbTx = ZBTxRequest(addr64, payload, sizeof(payload));
ZBTxStatusResponse txStatus = ZBTxStatusResponse();
int statusLed = 13;
int errorLed = 12;
void flashLed(int pin, int times, int wait) {
for (int i = 0; i < times; i++) {
digitalWrite(pin, HIGH);
delay(wait);
digitalWrite(pin, LOW);
if (i + 1 < times) {
delay(wait);
}
}
}
double Thermistor(int RawADC)
{
double Temp;
Temp = log(10000.0 * ((1024.0 / RawADC - 1)));
Temp = 1 / (0.001129148 + (0.000234125 + (0.0000000876741 * Temp * Temp )) * Temp );
Temp = Temp - 273.15; // Convert Kelvin to Celcius
//Temp = (Temp * 9.0)/ 5.0 + 32.0; // Convert Celcius to Fahrenheit
return Temp;
}
void setup() {
pinMode(statusLed, OUTPUT);
pinMode(errorLed, OUTPUT);
Serial.begin(9600);
}
void loop() {
float rawADC = analogRead(sensor);
float t = Thermistor (rawADC);
// check if returns are valid, if they are NaN (not a number) then something went wrong!
if (!isnan(t)) {
// convert temperature into a byte array and copy it into the payload array
u.fval = t;
for (int i=0;i<4;i++){
payload[i]=u.b[i];
}
u.fval = 100.33;
for (int i=0;i<4;i++){
payload[i+4]=u.b[i];
}
xbee.send(zbTx);
flashLed(statusLed, 1, 100); // flash TX indicator
// after sending a tx request, we expect a status response, wait up to half second for the status response
if (xbee.readPacket(500)) {
// got a response!
// should be a znet tx status
if (xbee.getResponse().getApiId() == ZB_TX_STATUS_RESPONSE) {
xbee.getResponse().getZBTxStatusResponse(txStatus);
// get the delivery status, the fifth byte
if (txStatus.getDeliveryStatus() == SUCCESS) {
// success. time to celebrate
flashLed(statusLed, 5, 50);
} else {
// the remote XBee did not receive our packet. is it powered on?
flashLed(errorLed, 3, 500);
}
}
} else if (xbee.getResponse().isError()) {
//nss.print("Error reading packet. Error code: ");
//nss.println(xbee.getResponse().getErrorCode());
} else {
// local XBee did not provide a timely TX Status Response -- should not happen
flashLed(errorLed, 1, 50);
}
}
delay(2000);
}
Raspberry Pi Code (Coordinator):
from xbee import ZigBee
import serial
import struct
import datetime
PORT = '/dev/ttyUSB0'
BAUD_RATE = 9600
def hex(bindata):
return ''.join('%02x' % ord(byte) for byte in bindata)
# Open serial port
ser = serial.Serial(PORT, BAUD_RATE)
# Create API object
xbee = ZigBee(ser,escaped=True)
# Continuously read and print packets
while True:
try:
response = xbee.wait_read_frame()
sa = hex(response['source_addr_long'])
rf = hex(response['rf_data'])
obj = createObject(response)
obj.createPacket()
print ("Temperature: %.2f" % obj.packet['temperature'],
"Humidity: %.2f" % obj.packet['humidity'],
"Source Address: 0x%s" % obj.packet['sourceAddressShort'],
"Timestamp: %s" % obj.packet['timestamp'].isoformat())
except KeyboardInterrupt:
break
ser.close()
class createObject:
def __init__(self, response):
self.sourceAddrLong = hex(response['source_addr_long'])
self.rfData = hex(response['rf_data'])
self.sourceAddrShort = hex(response['source_addr_long'][4:])
self.options = response.pop('options')
self.frameType = response['id']
self.temperature = struct.unpack('f',response['rf_data'][0:4])[0]
self.humidity = struct.unpack('f',response['rf_data'][4:])[0]
self.dataLength = len(response['rf_data'])
self.packet={}
self.dateNow = datetime.datetime.utcnow()
self.packetJson=0
def createPacket(self):
self.packet.update({
'timestamp' : self.dateNow,
'temperature' : self.temperature,
'humidity' : self.humidity,
'dataLength' : self.dataLength,
'sourceAddressLong' : self.sourceAddrLong,
'sourceAddressShort' : self.sourceAddrShort,
'options' : self.options,
'frameType' : self.frameType
})
I have a few questions that I can't find answers to. I have looked almost everywhere but still have some confusions.
In the Arduino code, there is a portion of code at the end where it checks for the status response (I did not write the code, found it on the internet). When I set it up, my errorLED connected to pin 12 blinks once and looking into the code it means that the "local XBee did not provide a timely TX Status Response". My question is, do I have to send a response myself from the coordinator in python or is it generated automatically? If I have to do it myself, how would I do it? Because right now, there is not response. My setup works fine as I'm getting correct values on my Pi.
When I have more than one router, how would I handle it in the code? Would I keep sending sensor values after every 2 seconds from the arduino and loop through the address on Pi or is there another way that it is done usually? I'm very confused about it.
Right now, if I add more routers, they will keep sending out frames with sensor values and the coordinator reads them in a loop. How can I setup the system such that the coordinator sends a signal to each router and asks for the data and then the router replies with the data? Is it possible?

The Transmit Status frame happens automatically when the local XBee has confirmed delivery to the remote XBee. It's a low-level acknowledgement. My guess is that there's a problem in that code's logic. Maybe the response comes back after 500ms. The only way to tell would be to restructure the code to constantly poll the local XBee for frames, send the sensor status frame every two seconds, and keep track of how long it's been since the last successful Transmit Status frame comes in. And I'd recommend increasing the baud rate here as well, especially since the existing Arduino code isn't processing bytes as often as it should (e.g., 2 seconds of idling without reading the serial port).
It looks like the Raspberry Pi code is already set up to handle data from multiple devices inside the "Continuously read and print packets" loop. I would suggest configuring your XBee module to 115200bps and updating the Python code with the new value, so you're not limiting the data rate.
Your current design is easier to manage -- routers on the network will always report their sensor readings. I think you can even update the code to use 00000000-00000000 as the destination address, and the router will always send to the coordinator. But you could modify the Raspberry Pi code to keep a list of router MAC addresses (discovered via ATND node discovery), and send requests to them as needed. You'd need to modify the Arduino code to watch for inbound frames, and generate an outbound frame when a request comes in.
I'd recommend adding a second router to your setup without any code changes, and then just seeing how that works. From what I can tell, the Raspberry Pi will just print packets with different source addresses as they come in.