We Keep Coding

Python-CAN script receiving half of the expected CAN messages

I have written a Python script utilizing the Python-CAN library which records received CAN messages at a 1 second rate for 5 minutes, before logging all the messages into a file and exiting. The computer has a CAN module which is connecting to the CAN bus. (The other device on the bus is an engine) I communicate with it using the SocketCAN interface.
The test engine system that this computer is connected to is sending around 114 messages at what I believe is a 250kb baud rate. I am expecting to see 114 messages recorded in the file for each 1 second period, but instead I'm seeing about half that count. (~65 messages)
Could it be possible that the engine's ECU is set to a 500kb baud rate, and that's why I'm not getting the count I am expecting? I would think there would be no communication if the baud rates do not match, but I do not have physical access to the system because I'm sending the script remotely through an OTA update and not running it myself. (The device is headless, but is setup to run the script on startup) I just see the log files that are generated.
Here is the python code:
(A note, I have code parsing the received messages into the contained signals, but I did not include this code here because it happens at the end, and it is not relevant)
class logging:
def __init__(self):
#Dictionary to hold received CAN messages
self.message_Dict = {}
#List to hold queued dictionaries
self.message_Queue = []
#A "filters" object is also created here, but I did not include it
#I have verified the filters are correct on my test system
def main(self):
#Record the current time
currentTime = datetime.datetime.now()
#Record the overall start time
startTime = datetime.datetime.now()
#Record the iteration start time
lastIterationStartTime = currentTime
#Create the CanBus that will be used to send and receive CAN msgs from the MCU
canbus = can.interfaces.socketcan.SocketcanBus(channel='can0', bitrate=250000)
#These filters are setup correctly, because all the messages come through
#on my test system, but I did not include them here
canbus.set_filters(self.Filters)
# Creating Listener filters and notifier
listener = can.Listener()
#Main loop
while 1:
#create a variable to hold received data
msg2 = canbus.recv()
#Record the current time
currentTime = datetime.datetime.now()
#If a valid message is detected
if(msg2 != None):
if(len(msg2.data) > 0):
try:
#Save the message data into a queue (will be processed later)
self.message_Dict[msg2.arbitration_id] = msg2.data
except:
print("Error in storing CAN message")
#If 1 second has passed since the last iteration,
#add the dictionary to a new spot in the queue
if((currentTime - lastIterationStartTime) >= datetime.timedelta(seconds=1)):
#Add the dictionary with messages into the queue for later processing
messageDict_Copy = self.message_Dict.copy()
self.message_Queue.append(messageDict_Copy)
print("Number of messages in dictionary: " + str(len(self.message_Dict)) + "
Number of reports in queue: " + str(len(self.message_Queue)))
#Clear the dictionary for new messages for every iteration
self.message_Dict.clear()
#Record the reset time
lastIterationStartTime = datetime.datetime.now()
#Once 5 minutes of data has been recorded, write to the file
if((currentTime - startTime) > datetime.timedelta(minutes=5)):
#Here is where I write the data to a file. This is too long to include
#Clear the queue
self.message_Queue = []
#Clear the dictionary for new messages for every iteration
self.message_Dict.clear()
#When the script is run, execute the Main method
if __name__ == '__main__':
mainClass = logging()
mainClass.main()
I appreciate any ideas or input you have. Thank you

In my experience, most of the engine's ECU usually uses 250kb, but the newest ones are using 500kb. I would suggest you too also try the both.
Also, the messages will only come to the bus if they have been sent, it seems silly but for example a truck, if you don't step on the accelerator the messages referring to the accelerator will not appear. So, maybe you need to check if all components are being using as you expect. The lib of can-utils has a 'Can sniffer' that can also help you.
I suggest you to use 'can-utils' to help you in that. It is a powerful tool to can analyses.
Did you try to loop the baudrate? Maybe can also help to find another.

python threads referencing the same class at different memory addresses

Im not quite sure what to classify this problem as, but as far as i can tell the connhandler thread should be able to write to the dictionary stored in the queuemanager class that is defined in hostmain, however when i print it out from the terminal prompt it comes up as empty, and when the thread and hostmain print the class object they have different memory addresses. This is my first project with multithreading and socketserver so i may have done something wrong, but to my knowledge i cant see why the thread from socketserver is unable to access the same servermanager object from hostmain.
To clarify a bit this project is using the socketserver library with the threading mixin, and the the threading library for everything showed in the example.
Request handler class code as well as a pastebin with pared down versions of pertinent code, reproducing the issue:
https://pastebin.com/u/kadytoast/1/PPWfyCFT
import testhostmain as hmain
import multiprocessing as mp
def joinpacket(flag, data):
return f"{flag}{hmain.packetdelim}{data}"
def splitpacket(msg):
msg = msg.split(hmain.packetdelim)
flag, data = msg[0], msg[1]
return flag, data
class ConnHandler(soss.BaseRequestHandler):
def handle(self):
# request handler to spawn processes
print(hmain.servermanager, "in connhandler")
# collecting peritype and name
msg = self.request.recv(hmain.buffer).decode("utf-8")
flag, data = splitpacket(msg)
data = data.split(hmain.datadelim)
self.peritype, self.macid = data[0], data[1]
#print(data)
# checks the header flag
if flag == hmain.newconflag:
hmain.servermanager.procdict[self.macid] = "connected"
print(hmain.servermanager.procdict, "from connhandler")
in this image you can see the two class printouts with their memory addresses, i would assume they should be the same but im not sure on that, however the dictionary printout from connhandler is correct but doesnt show up from the current process dictionary printout.
is there some conflict trying to use the threading mixin as well as seperately called threads?
Thankyou for getting this far if you did, if i need to provide anymore information i am happy to oblige. Thankyou again!
Edit: i expect i may have to add usage of the threading.lock, but nothing is writing to the dictionary at all so i dont expect that to be the issue in this case

is it possible to pass data from one python program to another python program? [duplicate]

Is it possible -- other than by using something like a .txt/dummy file -- to pass a value from one program to another?
I have a program that uses a .txt file to pass a starting value to another program. I update the value in the file in between starting the program each time I run it (ten times, essentially simultaneously). Doing this is fine, but I would like to have the 'child' program report back to the 'mother' program when it is finished, and also report back what files it found to download.
Is it possible to do this without using eleven files to do it (that's one for each instance of the 'child' to 'mother' reporting, and one file for the 'mother' to 'child')? I am talking about completely separate programs, not classes or functions or anything like that.
To operate efficently, and not be waiting around for hours for everything to complete, I need the 'child' program to run ten times and get things done MUCH faster. Thus I run the child program ten times and give each program a separate range to check through.
Both programs run fine, I but would like to get them to run/report back and forth with each other and hopefully not be using file 'transmission' to accomplish the task, especially on the child-mother side of the transferring of data.
'Mother' program...currently
import os
import sys
import subprocess
import time
os.chdir ('/media/')
#find highest download video
Hival = open("Highest.txt", "r")
Histr = Hival.read()
Hival.close()
HiNext = str(int(Histr)+1)
#setup download #1
NextVal = open("NextVal.txt","w")
NextVal.write(HiNext)
NextVal.close()
#call download #1
procs=[]
proc=subprocess.Popen(['python','test.py'])
procs.append(proc)
time.sleep(2)
#setup download #2-11
Histr2 = int(Histr)/10000
Histr2 = Histr2 + 1
for i in range(10):
Hiint = str(Histr2)+"0000"
NextVal = open("NextVal.txt","w")
NextVal.write(Hiint)
NextVal.close()
proc=subprocess.Popen(['python','test.py'])
procs.append(proc)
time.sleep(2)
Histr2 = Histr2 + 1
for proc in procs:
proc.wait()
'Child' program
import urllib
import os
from Tkinter import *
import time
root = Tk()
root.title("Audiodownloader")
root.geometry("200x200")
app = Frame(root)
app.grid()
os.chdir('/media/')
Fileval = open('NextVal.txt','r')
Fileupdate = Fileval.read()
Fileval.close()
Fileupdate = int(Fileupdate)
Filect = Fileupdate/10000
Filect2 = str(Filect)+"0009"
Filecount = int(Filect2)
while Fileupdate <= Filecount:
root.title(Fileupdate)
url = 'http://www.yourfavoritewebsite.com/audio/encoded/'+str(Fileupdate)+'.mp3'
urllib.urlretrieve(url,str(Fileupdate)+'.mp3')
statinfo = os.stat(str(Fileupdate)+'.mp3')
if statinfo.st_size<10000L:
os.remove(str(Fileupdate)+'.mp3')
time.sleep(.01)
Fileupdate = Fileupdate+1
root.update_idletasks()
I'm trying to convert the original VB6 program over to Linux and make it much easier to use at the same time. Hence the lack of .mainloop being missing. This was my first real attempt at anything in Python at all hence the lack of def or classes. I'm trying to come back and finish this up after 1.5 months of doing nothing with it mostly due to not knowing how to. In research a little while ago I found this is WAY over my head. I haven't ever did anything with threads/sockets/client/server interaction so I'm purely an idiot in this case. Google anything on it and I just get brought right back here to stackoverflow.
Yes, I want 10 running copies of the program at the same time, to save time. I could do without the gui interface if it was possible for the program to report back to 'mother' so the mother could print on the screen the current value that is being searched. As well as if the child could report back when its finished and if it had any file that it downloaded successfully(versus downloaded and then erased due to being to small). I would use the successful download information to update Highest.txt for the next time the program got ran.
I think this may clarify things MUCH better...that or I don't understand the nature of using server/client interaction:) Only reason time.sleep is in the program was due to try to make sure that the files could get written before the next instance of the child program got ran. I didn't know for sure what kind of timing issue I may run into so I included those lines for safety.

This can be implemented using a simple client/server topology using the multiprocessing library. Using your mother/child terminology:
server.py
from multiprocessing.connection import Listener
# client
def child(conn):
while True:
msg = conn.recv()
# this just echos the value back, replace with your custom logic
conn.send(msg)
# server
def mother(address):
serv = Listener(address)
while True:
client = serv.accept()
child(client)
mother(('', 5000))
client.py
from multiprocessing.connection import Client
c = Client(('localhost', 5000))
c.send('hello')
print('Got:', c.recv())
c.send({'a': 123})
print('Got:', c.recv())
Run with
$ python server.py
$ python client.py

When you talk about using txt to pass information between programs, we first need to know what language you're using.
Within my knowledge of Java and Python achi viable despite laborious depensendo the amount of information that wants to work.
In python, you can use the library that comes with it for reading and writing txt and schedule execution, you can use the apscheduler.

Python, pyserial program for communicating with Zaber TLSR300B

I'm still new here so I apologize if I make any mistakes or if my question is not specific enough, please correct me! I'm working on a program for controlling two Zaber TLSR300B linear motion tracks in a laser lab via a serial connection. I have no problems communicating with them using pyserial, I've been able to write and read data no problem. My questions is more about how to structure my program to achieve the desired functionality (I have very little formal programming training).
What I would like the program to do is provide a number of methods which allow the user to send commands to the tracks which then return what the track responds. However, I do not want the program to hang while checking for responses, so I can't just write methods with write commands followed by a read command. For some commands, the tracks respond right away (return ID, return current position, etc.) but for others the tracks respond once the requested action has been performed (move to location, move home, etc.). For example, if a move_absolute command is sent the track will move to the desired position and then send a reply with its new position. Also, the tracks can be moved manually with a physical knob, which causes them to continuously send their current position as they move (this is why I need to continuously read the serial data).
I've attached code below where I implement a thread to read data from the serial port when there is data to read and put it in a queue. Then another thread takes items from the queue and handles them, modifying the appropriate values of the ZaberTLSR300B objects which store the current attributes of each track. The methods at the bottom are some of the methods I'd like the user to be able to call. They simply write commands to the serial port, and the responses are then picked up and handled by the continuously running read thread.
The main problem I run into is that those methods will have no knowledge of what the track responds so they can't return the reply, and I can't think of a way to fix this. So I can't write something like:
newPosition = TrackManager.move_absolute(track 1, 5000)
or
currentPosition = TrackManager.return_current_position(track 2)
which is in the end what I want the user to be able to do (also since I'd like to implement some sort of GUI on top of this in the future).
Anyways, am I going about this in the right way? Or is there a cleaner way to implement this sort of behavior? I am willing to completely rewrite everything if need be!
Thanks, and let me know if anything is unclear.
Zaber TLSR300B Manual (if needed): http://www.zaber.com/wiki/Manuals/T-LSR
CODE:
TRACK MANAGER CLASS
import serial
import threading
import struct
import time
from collections import deque
from zaberdevices import ZaberTLSR300B
class TrackManager:
def __init__(self):
self.serial = serial.Serial("COM5",9600,8,'N',timeout=None)
self.track1 = ZaberTLSR300B(1, self.serial)
self.track2 = ZaberTLSR300B(2, self.serial)
self.trackList = [self.track1, self.track2]
self.serialQueue = deque()
self.runThread1 = True
self.thread1 = threading.Thread(target=self.workerThread1)
self.thread1.start()
self.runThread2 = True
self.thread2 = threading.Thread(target=self.workerThread2)
self.thread2.start()
def workerThread1(self):
while self.runThread1 == True:
while self.serial.inWaiting() != 0:
bytes = self.serial.read(6)
self.serialQueue.append(struct.unpack('<BBl', bytes))
def workerThread2(self):
while self.runThread2 == True:
try:
reply = self.serialQueue.popleft()
for track in self.trackList:
if track.trackNumber == reply[0]:
self.handleReply(track, reply)
except:
continue
def handleReply(self, track, reply):
if reply[1] == 10:
track.update_position(reply[2])
elif reply[1] == 16:
track.storedPositions[address] = track.position
elif reply[1] == 20:
track.update_position(reply[2])
elif reply[1] == 21:
track.update_position(reply[2])
elif reply[1] == 60:
track.update_position(reply[2])
def move_absolute(self, trackNumber, position):
packet = struct.pack("<BBl", trackNumber, 20, position)
self.serial.write(packet)
def move_relative(self, trackNumber, distance):
packet = struct.pack("<BBl", trackNumber, 21, distance)
self.serial.write(packet)
def return_current_position(self, trackNumber):
packet = struct.pack("<BBl", trackNumber, 60, 0)
self.serial.write(packet)
def return_stored_position(self, trackNumber, address):
packet = struct.pack("<BBl", trackNumber, 17, address)
self.serial.write(packet)
def store_current_position(self, trackNumber, address):
packet = struct.pack("<BBl", trackNumber, 16, address)
self.serial.write(packet)
zaberdevices.py ZaberTLSR300B class
class ZaberTLSR300B:
def __init__(self, trackNumber, serial):
self.trackNumber = trackNumber
self.serial = serial
self.position = None
self.storedPositions = []
def update_position(self, position):
self.position = position

There is an option on the controllers to disable all of the replies that don't immediately follow a command. You can do this by enabling bits 0 and 5 of the Device Mode setting. These bits correspond to 'disable auto-replies' and 'disable manual move tracking'. Bit 11 is enabled by default, so the combined value to enable these bits is 2081.
My recommendation would be to disable these extra responses so that you can then rely on the predictable and reliable command->response model. For example to move a device you would send the move command, but never look for a response from that command. To check whether the movement has completed, you could either use the Return Current Position command and read the position response, or use the Return Status (Cmd_54) command to check whether the device is busy (i.e. moving) or idle.

So, the way I went about doing this when a multithreaded application needed to share the same data was implementing my own "locker". Basically your threads would call this ChekWrite method, it would return true or false, if true, it would set the lock to true, then use the shared resource by calling another method inside of it, after complete, it would set the lock to false. If not, it would wait some time, then try again. The check write would be its own class. You can look online for some Multithreading lock examples and you should be golden. Also, this is all based on my interpretation of your code and description above... I like pictures.
Custom Locks Threading python
/n/r
EDIT
My fault. When your ListenerThread obtains data and needs to write it to the response stack, it will need to ask permission. So, it will call a method, CheckWrite, which is static, This method will take in the data as a parameter and return a bit or boolean. Inside this function, it will check to see if it is in use, we will call it Locked. If it is locked, it will return false, and your listener will wait some time and then try again. If it is unlocked. It will set it to locked, then will proceed in writing to the response stack. Once it is finished, it will unlock the method. On your response stack, you will have to implement the same functionality. It slipped my mind when i created the graphic. Your main program, or whatever wants to read the data, will need to ask permission to read from, lock the Locker value and proceed to read the data and clear it. Also, you can get rid of the stack altogether and have a method inside that write method that goes ahead and conducts logic on the data. I would separate it out, but to each their own.

developing for modularity & reusability: how to handle While True loops?

I've been playing around with the pybluez module recently to scan for nearby Bluetooth devices. What I want to do now is extend the program to also find nearby WiFi client devices.
The WiFi client scanner will have need to have a While True loop to continually monitor the airwaves. If I were to write this as a straight up, one file program, it would be easy.
import ...
while True:
client = scan()
print client['mac']
What I want, however, is to make this a module. I want to be able to reuse it later and, possible, have others use it too. What I can't figure out is how to handle the loop.
import mymodule
scan()
Assuming the first example code was 'mymodule', this program would simply print out the data to stdout. I would want to be able to use this data in my program instead of having the module print it out...
How should I code the module?

I think the best approach is going to be to have the scanner run on a separate thread from the main program. The module should have methods that start and stop the scanner, and another that returns the current access point list (using a lock to synchronize). See the threading module.

How about something pretty straightforward like:
mymodule.py
import ...
def scanner():
while True:
client = scan()
yield client['mac']
othermodule.py
import mymodule
for mac in mymodule.scanner():
print mac
If you want something more useful than that, I'd also suggest a background thread as #kindall did.

Two interfaces would be useful.
scan() itself, which returned a list of found devices, such that I could call it to get an instantaneous snapshot of available bluetooth. It might take a max_seconds_to_search or a max_num_to_return parameter.
A "notify on found" function that accepted a callback. For instance (maybe typos, i just wrote this off the cuff).
def find_bluetooth(callback_func, time_to_search = 5.0):
already_found = []
start_time = time.clock()
while 1:
if time.clock()-start_time > 5.0: break
found = scan()
for entry in found:
if entry not in already_found:
callback_func(entry)
already_found.append(entry)
which would be used by doing this:
def my_callback(new_entry):
print new_entry # or something more interesting...
find_bluetooth(my_callback)

If I get your question, you want scan() in a separate file, so that it can be reused later.
Create utils.py
def scan():
# write code for scan here.
Create WiFi.py
import utils
def scan_wifi():
while True:
cli = utils.scan()
...
return