We Keep Coding

How are virtual COM/Serial ports managed by operating system?

I'm currenty working on an interface between arduino and raspberry pi. The arduino is connected via usb cable to the Rasbperry Pi. The Arduino has an LED connected to a digital Pin that may be turned on and off by sending specified bytes over the USB serial connection. The Raspberry Pi then runs the following python script:
import serial
import time
arduino = serial.Serial('/dev/ttyACM0', baudrate=115200, timeout=.1)
time.sleep(2)
start = time.time()
for _ in range(10):
arduino.write(bytes([0x58])) //Turns LED on (on Arduino)
print(time.time()-start,'on')
time.sleep(.5)
arduino.write(bytes([0x18])) //Turns LED off (on Arduino)
print(time.time()-start,'off')
time.sleep(.5)
arduino.write(spam_the_COM_port) # will let the arduino start sending
# infinite amout of bytes
print('started spamming')
for _ in range(10):
arduino.write(turn_led_on) #Actual byte is 0x58
print(time.time()-start,'on')
time.sleep(.5)
arduino.write(turn_led_off) #Actual byte is 0x18
print(time.time()-start,'off')
time.sleep(.5)
So I'm switching on and off an LED that is connected to an arduino by sending it the specified bytes: 0x58 and 0x18 The Arduino runs code that looks like this:
bool spamming = false;
void setup()
{
Serial.begin(115200);
pinMode(24, OUTPUT);
}
void loop()
{
while(Serial.available() > 0)
{
unsigned char x = Serial.read();
if (x == 0x58)
{
digitalWrite(24, HIGH); //Turn on LED
}
else if (x == 0x18)
{
digitalWrite(24, LOW); //Turn off LED
}
else if (x == 0x00)
{
spamming = true;
}
}
if (spamming)
{
if (Serial.avialiableForWrite() > 0) Serial.write(0x01);
}
}
I get the expected result (which is just the console printing on and off 10 times (on the raspberry side) and the Arduino actually turning the LED on and off 10 times in a row.
I then send a command that will make the Arduino start sending an infinite amount of bytes to the Raspberry Pi whilst still checking the Serial incoming buffer for new commands. After that I repeat the same process as before and try to send the LED switching command.
Now the python console output stays the same (runs without any error raising), yet the program blocks at each arduino.write() command for a long time (around 10s or longer) even though there is a timeout specified (0.1s). After blocking, the console prints as usual and the LED on the arduino switches as usual. I just added timesamps to the python console output and the results look like this:
2.002582311630249 on
2.502887725830078 off
3.003848075866699 on
3.5050783157348633 off
4.005117177963257 on
4.5058839321136475 off
5.006282806396484 on
5.50661826133728 off
6.008156776428223 on
6.509922504425049 off
7.010948181152344 on
7.512096166610718 off
8.012341737747192 on
8.513236999511719 off
9.013462543487549 on
9.514604091644287 off
10.015231370925903 on
10.515642404556274 off
11.01583743095398 on
11.516374349594116 off
started spamming
12.018024444580078 on
12.518478155136108 off
13.01881217956543 on
13.519550085067749 off
28.621582508087158 on
44.68230104446411 off
57.40802550315857 on
63.064852714538574 off
65.58953928947449 on
68.61984610557556 off
77.50833535194397 on
96.19384908676147 off
102.6581130027771 on
109.2223105430603 off
132.55511474609375 on
155.684588432312 off
171.44103860855103 on
185.78372859954834 off
199.51972341537476 on
216.58949422836304 off
So the problem must be on the raspberry pi. I tried the same code on my windows machine (plugging the arduino in there of course) and the same thing happened, which led to the conclusion that it must have something to do with the virtual COM/Serial port.
So now to the actual question: Why does it take so long for the serial.write() function to write something to the output buffer (on the pi) when the input buffer (on the pi) is beeing spammed by the arduino?
As far as I know, a UART Chip has two physically seperate buffers that are completely independent. Does a virtual COM port not have this independence of the two buffers? (Maybe just some allocated memory for both of them in a stack-heap kind of way?) Or is the problem that USB 1.0 and 2.0 are only half-duplex?
(And then lastly is there a way to fix the fucntion block?)
[update]
I managed to stop the function from blocking by adding
if arduino.inWaiting() > 1000: arduino.flushInput()
before every arduino.write() statement. I'm still interested in how the Serial/COM port is managed by the operating system, be it linux/raspbian or windows.

Python serial.write() not working for NodeMCU

I am fairly new to hardware. I want to control an LED light using NodeMCU and Python. I uploaded an Arduino code in nodeMCU and then used pyserial library to get the serial output. But when I try to give input to the port, it doesn't work. I don't know where the problem is.
Here is the arduino code:
int inputVal = 0;
const int ledPin = 5; //D1 pin of NodeMCU
void setup() {
Serial.begin(9600);
delay(100);
pinMode(ledPin, OUTPUT);
digitalWrite(ledPin, 0);
}
void loop() {
while(Serial.available()>0){
inputVal = Serial.read();
}
Serial.println(inputVal);
if(inputVal==1){
digitalWrite(ledPin, HIGH);
Serial.println("LED is ON");
}
else{
digitalWrite(ledPin, LOW);
Serial.println("LED is OFF");
}
Serial.println("");
}
Here is the python code:
import serial
global ser
ser = serial.Serial("COM8", baudrate=9600, timeout=10,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS)
while(True):
ser.write(bytes(1))
line = ser.readline()
print(line.decode('utf8'))
The output in python comes out to be:
0
LED is OFF
0
LED is OFF
0
LED is OFF
and so on. The ser.write() function isn't writing the value as 1 on the serial port. When I change the value of inputVal in Arduino code, the LED turns on and the output on arduino serial monitor comes as 1 LED is ON, which implies that the circuit connection and Arduino code is working fine.
I also noticed that the COM port that I am using can work with either python or arduino at a time. After uploading the arduino code with inputVal=1, the LED turned on and arduino serial monitor started displaying (1 LED is ON). But, as soon as I ran the python code, the led turned off and the python output came out to be 0 LED is OFF. Please help me.
Also, is there a way for me to control NodeMCU totally by python, without using arduino code first?

the output from python is correct. bytes(integer) creates an array of provided size, all initialized to null in your case size = 1, bytes(1), so the output that you have is 0x00 if you try bytes(10) the out put will be b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'.
what you need to do is to change ser.write(bytes(1)) to ser.write(bytes('1',encoding= 'utf-8')) that should work

How to Read Data from Arduino with Raspberry pi via I2C

I have connected Raspberry pi 2 model B with arduino uno via Bi-Directional Level shifter.
Raspberry pi GND ---------- GND Arduino
3.3v ---------- 5v
SCL ---------- A5
SDA ---------- A4
Hope my I2C connection is correct ?
and my Arduino is connected to 8-Channel Relay Board.
Now I have written code in which I can control the Relay board by Raspberry pi. For ex if i Press '1' the Relay 1 goes high.
Now I want to send data back from arduino to raspberry pi in order to cross check if Relay 1 is high or not, if Relay 1 is high then it should send some data back to Raspberry pi or else not.
My Rpi code is
import smbus
import time
# for RPI version 1, use "bus = smbus.SMBus(0)"
bus = smbus.SMBus(1)
# This is the address we setup in the Arduino Program
address = 0x04
def writeNumber(value):
bus.write_byte(address, value)
# bus.write_byte_data(address, 0, value)
return -1
def readNumber():
number = bus.read_byte(address)
# number = bus.read_byte_data(address, 1)
return number
while True:
var = input("")
if not var:
continue
writeNumber(var)
number = readNumber()
My Arduino code:
#include <Wire.h>
#define SLAVE_ADDRESS 0x04
#define RELAY1 9
int number = 0;
int state = 0;
void setup() {
pinMode(RELAY1, OUTPUT);
Serial.begin(9600); // start serial for output
// initialize i2c as slave
Wire.begin(SLAVE_ADDRESS);
// define callbacks for i2c communication
Wire.onReceive(receiveData);
Wire.onRequest(sendData);
Serial.println("Ready!");
}
void loop() {
delay(100);
}
// callback for received data
void receiveData(int byteCount){
while(Wire.available()) {
number = Wire.read();
Serial.print("data received: ");
Serial.println(number);
if (number == 1){
if (state == 0){
digitalWrite(RELAY1, HIGH); // set the LED on
state = 1;
}
else{
digitalWrite(RELAY1, LOW); // set the LED off
state = 0;
}
}
}
}
// callback for sending data
void sendData(){
Wire.write(number);
}
Now if I type 1 and due to some loose connection Relay 1 doesn't goes high, So in this case I want the arduino to take data from relay board and send it to Raspberry pi every time.
It will be great if someone can explain also that how it works.
Hope I was able to explain the problem. I have done lots of research but was not able to find some answer.
I am a beginner in python so please help me.
Thanks in advance.

The problem is that you are doing too much inside receiveData, which is called from the Interrupt Service Routine of the I2C utility code, twi.c. You must handle the data quickly, and don't call any other routines that depend on interrupts being enabled (they are disabled during this ISR).
This means you can't call Serial.print, and you can't call any other Wire sending methods. Even calling millis() or micros() is discouraged, as they do take a fair amount of time, and they depend on the TIMER interrupts being handled.
Of course, you are free call Wire.available() and Wire.read(). Actually, byteCount tells you how many bytes are available, so you don't need to call Wire.available() again.
Essentially, your receivedData routine can read the data inside the routine if you're quick about processing it. Otherwise, you can only set a (volatile) flag and then watch for it in loop. From what I see in your sketch, you could do something like this:
// variables that allow signalling between receiveData ISR and loop
volatile bool newData = false;
volatile uint8_t state = false;
// callback for received data
void receiveData(int byteCount)
{
// Read all the bytes; only the last one changes the relay state
while (byteCount-- > 0)
number = Wire.read();
if (state != number) {
state = number;
newData = true;
}
}
// callback for sending data
void sendData(){
Wire.write(number);
}
void loop()
{
if (newData) {
newData = false; // clear the flag for next time
if (number == 1){
digitalWrite(RELAY1, HIGH); // set the LED on
} else {
digitalWrite(RELAY1, LOW); // set the LED off
}
Serial.print("data received: ");
Serial.println( number );
}
}
The delay in loop is unnecessary, and may cause problems if you add something else to loop.
The volatile keyword keeps the compiler from optimizing loop. Without that keyword, the test for newData in loop would disappear because the compiler thinks that newData doesn't change during loop. Why test it? volatile newData tells the compiler that newData can change at any time, like during the receiveData ISR.
And be sure to print the number in the rpi code, as pholtz suggested!

In arduino code
change sendData() function like this
void sendData(){
int relay_status;
relay_status=digitalRead(4);
Wire.write(relay_status);
}
Also in hardware connect one 4th digital pin(or any other free I/O pins) to relay input.
hope it helps:)

Ok, looks like a pretty good start. Two things I want to suggest here.
First, in your python program you should print number so that you can see it's value changing. It's storing your feedback from the Arduino, so you want to display that feedback to the screen. This is as simple as changing number = readNumber() to print readNumber().
Second, in your Arduino program, are you sure that calling Wire.read() returns what you think it does? It looks to me like read() returns a byte. Chances are that when you type 1 it's really being sent as '1', not 1. Char vs. Int. Make sense?
So you might want to check if(number == '1') instead. Just my 2¢.

Your i2c bus is not connected properly. Remove the level shifter, and add 4.7k pull-ups to the 3.3v vcc on scl and sda lines. I2c chips only drive the line low, and require external resistors to pull the line high. This lets you mix voltage levels on your i2c bus fairly easily. Then you can go back to figuring out what your code is doing.

Hanging python script with arduino. Need help simplifying things

Ok so what I'm trying to do is turn an LED on with one python script and off with another one. Now the problem I'm facing is my python script has to keep hanging for the LED to stay on. I can't figure out how to read something from serial, close the coms while leaving the LED on.
'g' is what I'm sending from the on python script and 'h' will be sent from the off python script.
The arduino:
void setup(){
Serial.begin(9600);
pinMode(13, OUTPUT);
Serial.write('g');
Serial.write('h');
}
void loop(){
if(Serial.read() == 'g' ){
digitalWrite(13, HIGH);
Serial.end();
}
if(Serial.read() == 'h' ){
digitalWrite(13, LOW);
Serial.end();
}
}
And the python
#! /usr/bin/python
## import the serial library
import serial
## Boolean variable that will represent
## whether or not the arduino is connected
connected = False
## open the serial port that your ardiono
## is connected to.
ser = serial.Serial("/dev/cu.wchusbserial1410", 9600)
## loop until the arduino is ready
while not connected:
serin = ser.read()
connected = True
ser.write("g")
while ser.read() == 'g':
ser.read()
## close the port
ser.close()
the 'while ser.read() parts at the bottom was just me messing about trying to figure out what I need but so far no such luck.
Thanks in advance!

In python code instead of using this serial command, simply use print command. Suppose you wanna send character g on the serial port then simply write:
print "g"
and it will be sent over to serial port. Worked for me while using Arduino YUN.

Thanks for the feedback. I used a different method and thought it would be a good idea to share the code incase anyone is interested in doing the same.
Python:
import serial
import time
arduino = serial.Serial('/dev/tty.wchusbserial1410', 9600)
time.sleep(0.1) # wait
print("initialising")
arduino.write('off') # turns LED off
print("LED OFF")
time.sleep(0.1) # wait
arduino.close() # close serial
This is the code used to turn the light off. If you want to turn it on, it's the same procedure but create another script replacing arduino.write('off') with arduino.write('on')
And Arduino:
int led = 13; // Pin 13
void setup()
{
pinMode(led, OUTPUT); // Set pin 13 as digital out
// Start up serial connection
Serial.begin(9600);
Serial.flush();
}
void loop()
{
String input = "";
// Read any serial input
while (Serial.available() > 0)
{
input += (char) Serial.read(); // Read in one char at a time
delay(5); // Delay for 5 ms so the next char has time to be received
}
if (input == "on")
{
digitalWrite(led, HIGH); // on
}
else if (input == "off")
{
digitalWrite(led, LOW); // off
}
}
The one problem with this script is after the serial coms close the light turns off. To fix this, I used a 10uF electrolytic capacitor between the ground and reset pin to keep the serial port open. (Please note: only put the cap in AFTER you've programmed the Arduino. If you need to reprogram, pull it out first.)

Random character at the beginning of serial.readline() in python and arduino

I get random characters when I perform a serial.readline(), sometimes it is other umbers and sometimes it is whole messages. The output should be "1,2" or "2,2"
Here are a couple of screenshots of the output of serial.readline().
I have tried to put a delay at before serial.readline() but it did not make a difference.
There is usually a strange character at the beginning:
I have also received strange messages:
There is a problem later on in the program that causes the program to hand because sometimes I just receive a blank line.
Is there a way to get consistent output from serial?
Here is the arduino code:
void setup() {
// initialize the LED pin as an output:
pinMode(ledPin, OUTPUT);
// initialize the pushbutton pin as an input:
pinMode(buttonPin, INPUT);
Serial.begin(9600);
}
void loop(){
// read the state of the pushbutton value:
buttonState = digitalRead(buttonPin);
// check if the pushbutton is pressed.
// if it is, the buttonState is HIGH:
if (buttonState == HIGH) {
// turn LED on:
digitalWrite(ledPin, HIGH);
Serial.println("1,2");
}
else {
// turn LED off:
digitalWrite(ledPin, LOW);
Serial.println("2,2");
}
}
And here is the python code:
ser = serial.Serial('/dev/ttyUSB0', 9600)
line=ser.readline()
coord= line.strip()
print coord
EDIT:
I tried putting ser.flushInput() after the ser.open() and I get the same output.

Have you flushed the serial buffer
ser.flushInput()

I've been having the same issue when interfacing between pyserial and Arduino. This may be an old post, but in case someone else is having the same trouble, I remedied my problem by inserting:
ser.flushInput()
...right before my ser.readline() call.

How to Read serial data from an Arduino in Linux
Nothing fancy here. I’m getting the current port configuration, setting the input/output speed to 9600 baud, setting the data expectations to be 8 bits per word, with no parity bit and no stop bit, setting canonical mode, and then committing those changes back to the serial port.
If I am not mistaken you have to change the mentioned settings when connecting through serial port.
You do not mention it, but I guess you are using the pySerial library. Regardless you can use it with the correct settings to connect through serial. The constructor of the API allows all the parameters which are noted below:
Pyserial Library
I have not tested this approach as I have had a similar problem in C not Python.

What happens if you break it down to something simpler?
On the Arduino:
void setup()
{
Serial.begin(9600);
}
int i = 0;
void loop()
{
delay(1000);
Serial.print("Hello World ");
Serial.println(i);
i++;
}
And in the Python REPL...
import serial
ser = serial.Serial('/dev/ttyUSB0', 9600)
while(1):
ser.readline()
The first behavior you should observe is that when you initiate the serial connection, the Arduino reboots. This takes a few seconds.
The second behavior you should observe is that (if you're typing this in the REPL slowly) when the while(1) loop begins, you get all of the serial data that had accumulated since you initiated the serial connection. It takes me a few seconds to type all that out, so when I hit Enter after ser.readline() I see:
'Hello World 1\r\n'
'Hello World 2\r\n'
'Hello World 3\r\n'
I only mention this to make sure you're familiar with two things that burned me a bit the last time I messed with serial communication to an Arduino: it needs time to reboot after connecting, and the serial data buffers. readline() will only give you one line in the buffer - not everything that's in the buffer.
Is it possible you're trying to sent/receive data before it's done rebooting? What about button bounce - could a dozen high/low state detections be messing something up?