We Keep Coding

How to properly read J1939 messages from .asc file with cantools?

I'm trying to create a CAN logs converter from .asc files to .csv files (in human readable form). I'm somewhat successful. My code works fine with almost any database but j1939.dbc.
The thing is, that if I print out the messages read from the dbc file, I can see that the messages from j1939.dbc are read into the database. But it fails to find any of those messages in the processed log file. At the same time I can read the same file using Vector CANalyzer with no issues.
I wonder why this may happed and why it only affects the j1939.dbc and not the others.
I suspect that maybe the way I convert those messages is wrong because it never goes by the if msg_id in database: line (and as mentioned above, those messages are certainly there because Vector CANalyzer works fine with them).
EDIT: I realized that maybe the problem is not cantools but python-can package, maybe the can.ASCReader() doeasn't do well with j1939 frames and omits them? I'm gonna investigate myself but I hope someone better at coding will help.
import pandas as pd
import can
import cantools
import time as t
from tqdm import tqdm
import re
import os
from binascii import unhexlify
dbcs = [filename.split('.')[0] for filename in os.listdir('./dbc/') if filename.endswith('.dbc')]
files = [filename.split('.')[0] for filename in os.listdir('./asc/') if filename.endswith('.asc')]
start = t.time()
db = cantools.database.Database()
for dbc in dbcs:
with open(f'./dbc/{dbc}.dbc', 'r') as f:
db.add_dbc(f)
f_num = 1
for fname in files:
print(f'[{f_num}/{len(files)}] Parsing data from file: {fname}')
log=can.ASCReader(f'./asc/{fname}.asc')
entries = []
all_msgs =[]
message = {'Time [s]': ''}
database = list(db._frame_id_to_message.keys())
print(database)
lines = sum(1 for line in open(f'./asc/{fname}.asc'))
msgs = iter(log)
try:
for msg, i in zip(msgs, tqdm(range(lines))):
msg = re.split("\\s+", str(msg))
timestamp = round(float(msg[1]), 0)
msg_id = int(msg[3], 16)
try:
data = unhexlify(''.join(msg[7:15]))
except:
continue
if msg_id in database:
if timestamp != message['Time [s]']:
entries.append(message.copy())
message.update({'Time [s]': timestamp})
message.update(db.decode_message(msg_id, data))
except ValueError:
print('ValueError')
df = pd.DataFrame(entries[1:])
duration = t.time() - start
df.to_csv(f'./csv/{fname}.csv', index=False)
print(f'DONE IN {int(round(duration, 2)//60)}min{round(duration % 60, 2)}s!\n{len(df.columns)} signals extracted!')
f_num += 1

class can.ASCReader(file, base=’hex’)
Bases: can.io.generic.BaseIOHandler
Iterator of CAN messages from a ASC logging file. Meta data (comments, bus statistics, J1939 Transport
Protocol messages) is ignored.
Might answer your question...

counter on for loop, reporting 1

I am simply trying to add a counter around my for loop, to count how many files are in my directory per qualification of the contains 'VCCS'... the logic is working for the iterating, as it's iterating through my directory the amount of times I have a file in... however my counter keeps reporting 1. Relevant lines are files_in_directory market with comment # here.
I am getting this warning in PyLint: Constant name "files_in_directory" doesn't conform to UPPER_CASE naming stylepylint(invalid-name) , on files_in_directory = 0
I've tried moving the set 0, above the for and try, any thoughts?
if __name__ == "__main__":
try:
currentDT = datetime.datetime.now()
files_in_directory = 0 # here
for filename in os.listdir(config.DIRECTORY_LOCATION):
if filename.__contains__('VCCS'):
old_stdout = sys.stdout
log_file = open("./logs/metrics.log","w")
sys.stdout = log_file
files_in_directory += 1 # here
PENDING_RECORDS = FindPendingRecords().get_excel_data()
# Do operations on PENDING_RECORDS
# Reads excel to map data from excel to vital
MAP_DATA = FindPendingRecords().get_mapping_data()
# Configures Driver
VITAL_ENTRY = VitalEntry()
# Start chrome and navigate to vital website
VITAL_ENTRY.instantiate_chrome()
# Begin processing Records
VITAL_ENTRY.process_records(PENDING_RECORDS, MAP_DATA)
print(f"Date: ")
print (str(currentDT))
print(f"Files in Directory #{files_in_directory}") # here
sys.stdout = old_stdout
log_file.close()
except Exception as exc:
# print(exc)
raise

Note: This is to take place of many comments in the interest of readability
Your question as it sits is not an MCVE. To make it more succinct and to identify the exact cause:
import os
if __name__ == "__main__":
# Remove try block, just let it raise an error
my_counter = 0
for file in os.listdir("some_directory"):
# you don't need to call the __contains__ method
# as the 'in' keyword will invoke that for you
if "VCCS" in file:
# increment your counter first
my_counter += 1
print(file, my_counter)
Now there is no question what is modifying my_counter, and this will print out the file you are looking at, alongside the counter.
Once you've ironed out that behavior, you can start adding in your other functions
import os
if __name__ == "__main__":
# Remove try block, just let it raise an error
my_counter = 0
for file in os.listdir("some_directory"):
if 'VCCS' in file:
my_counter += 1
print(my_counter, file)
# Add functions back in one by one
PENDING_RECORDS = FindPendingRecords().get_excel_data()
Continue this process until you identify what is causing your behavior. As it stands, I don't see anything explicit that could be overwriting that counter variable, so I suspect that either A) the code you posted does not reflect what is being run or B) you are modifying/resetting files_in_directory somewhere else in the module.
Suggested Edit:
I would recommend you add in the other code from your module to see what's going on. That way we have a clearer picture of what's happening as your code runs

So first thing first: you're sure you're passing the right directory, and that this directory contains indeed more than one file containing VCCS in their filename ?
I would also try to run this code without the Try/Except block, to see if you don't get some error after your first increment.
LMK what you get, I hope this helps.

Python - Put a list in a threading module

I want to put a list into my threading script, but I am facing a problem.
Contents of list file (example):
http://google.com
http://yahoo.com
http://bing.com
http://python.org
My script:
import codecs
import threading
import sys
import requests
from time import time as timer
from timeout import timeout
import time
try:
with codecs.open(sys.argv[1], mode='r', encoding='ascii', errors='ignore') as iiz:
iiz=iiz.read().splitlines()
except IOError:
pass
oz = list(iiz)
def nnn(url):
hzz = {'param1': sys.argv[2], 'param2': sys.argv[3]}
po = requests.post(url,data=hzz)
if po:
print("ok \n")
if __name__ == '__main__':
threads = []
for i in range(1):
t = threading.Thread(target=nnn, args=(oz,))
threads.append(t)
t.start()

Can you please clarify what elaborate on exactly what you're trying to achieve.
I'm guessing that you're trying to request urls to load into a web browser or the terminal...
Also you shouldn't need to put the urls into a list because when you opened up the file containing the urls, it automatically sorted it into a list. So in other words, the contents in iiz are already in the list format.
Personally, I haven't worked much with the modules you're using (apart from time), but I'll try my best to help you and hopefully other users will try and help you too.

When working with a named pipe is there a way to do something like readlines()

Overall Goal: I am trying to read some progress data from a python exe to update the progress of the exe in another application
I have a python exe that is going to do some stuff, I want to be able to communicate the progress to another program. Based on several other Q&A here I have been able to have my running application send progress data to a named pipe using the following code
import win32pipe
import win32file
import glob
test_files = glob.glob('J:\\someDirectory\\*.htm')
# test_files has two items a.htm and b.htm
p = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
win32pipe.PIPE_ACCESS_DUPLEX,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
1,65536,65536,300,None)
# the following line is the server-side function for accepting a connection
# see the following SO question and answer
""" http://stackoverflow.com/questions/1749001/named-pipes-between-c-sharp-and-python
"""
win32pipe.ConnectNamedPipe(p, None)
for each in testFiles:
win32file.WriteFile(p,each + '\n')
#send final message
win32file.WriteFile(p,'Process Complete')
# close the connection
p.close()
In short the example code writes the path of the each file that was globbed to the NamedPipe - this is useful and can be easily extended to more logging type events. However, the problem is trying to figure out how to read the content of the named pipe without knowing the size of each possible message. For example the first file could be named J:\someDirectory\a.htm, but the second could have 300 characters in the name.
So far the code I am using to read the contents of the pipe requires that I specify a buffer size
First establish the connection
file_handle = win32file.CreateFile("\\\\.\\pipe\\wfsr_pipe",
win32file.GENERIC_READ | win32file.GENERIC_WRITE,
0, None,
win32file.OPEN_EXISTING,
0, None)
and then I have been playing around with reading from the file
data = win32file.ReadFile(file_handle,128)
This generally works but I really want to read until I hit a newline character, do something with the content between when I started reading and the newline character and then repeat the process until I get to a line that has Process Complete in the line
I have been struggling with how to read only until I find a newline character (\n). I basically want to read the file by lines and based on the content of the line do something (either display the line or shift the application focus).
Based on the suggestion provided by #meuh I am updating this because I think there is a dearth of examples, guidance in how to use pipes
My server code
import win32pipe
import win32file
import glob
import os
p = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
win32pipe.PIPE_ACCESS_DUPLEX,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
1,65536,65536,300,None)
# the following line is the server-side function for accepting a connection
# see the following SO question and answer
""" http://stackoverflow.com/questions/1749001/named-pipes-between-c-sharp-and-python
"""
win32pipe.ConnectNamedPipe(p, None)
for file_id in glob.glob('J:\\level1\\level2\\level3\\*'):
for filer_id in glob.glob(file_id + os.sep + '*'):
win32file.WriteFile(p,filer_id)
#send final message
win32file.WriteFile(p,'Process Complete')
# close the connection
p.close() #still not sure if this should be here, I need more testing
# I think the client can close p
The Client code
import win32pipe
import win32file
file_handle = win32file.CreateFile("\\\\.\\pipe\\wfsr_pipe",
win32file.GENERIC_READ |
win32file.GENERIC_WRITE,
0, None,win32file.OPEN_EXISTING,0, None)
# this is the key, setting readmode to MESSAGE
win32pipe.SetNamedPipeHandleState(file_handle,
win32pipe.PIPE_READMODE_MESSAGE, None, None)
# for testing purposes I am just going to write the messages to a file
out_ref = open('e:\\testpipe.txt','w')
dstring = '' # need some way to know that the messages are complete
while dstring != 'Process Complete':
# setting the blocksize at 4096 to make sure it can handle any message I
# might anticipate
data = win32file.ReadFile(file_handle,4096)
# data is a tuple, the first position seems to always be 0 but need to find
# the docs to help understand what determines the value, the second is the
# message
dstring = data[1]
out_ref.write(dstring + '\n')
out_ref.close() # got here so close my testfile
file_handle.close() # close the file_handle

I don't have windows but looking through the api it seems you should convert
your client to message mode by adding after the CreateFile() the call:
win32pipe.SetNamedPipeHandleState(file_handle,
win32pipe.PIPE_READMODE_MESSAGE, None, None)
then each sufficiently long read will return a single message, ie what the other wrote in a single write. You already set PIPE_TYPE_MESSAGE when you created the pipe.

You could simply use an implementation of io.IOBase that would wrap the NamedPipe.
class PipeIO(io.RawIOBase):
def __init__(self, handle):
self.handle = handle
def read(self, n):
if (n == 0): return ""
elif n == -1: return self.readall()
data = win32file.ReadFile(self.file_handle,n)
return data
def readinto(self, b):
data = self.read(len(b))
for i in range(len(data)):
b[i] = data[i]
return len(data)
def readall(self):
data = ""
while True:
chunk = win32file.ReadFile(self.file_handle,10240)
if (len(chunk) == 0): return data
data += chunk
BEWARE : untested, but it should work after fixing the eventual typos.
You could then do:
with PipeIO(file_handle) as fd:
for line in fd:
# process a line

You could use the msvcrt module and open to turn the pipe into a file object.
Sending code
import win32pipe
import os
import msvcrt
from io import open
pipe = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
win32pipe.PIPE_ACCESS_OUTBOUND,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
1,65536,65536,300,None)
# wait for another process to connect
win32pipe.ConnectNamedPipe(pipe, None)
# get a file descriptor to write to
write_fd = msvcrt.open_osfhandle(pipe, os.O_WRONLY)
with open(write_fd, "w") as writer:
# now we have a file object that we can write to in a standard way
for i in range(0, 10):
# create "a\n" in the first iteration, "bb\n" in the second and so on
text = chr(ord("a") + i) * (i + 1) + "\n"
writer.write(text)
Receiving code
import win32file
import os
import msvcrt
from io import open
handle = win32file.CreateFile(r"\\.\pipe\wfsr_pipe",
win32file.GENERIC_READ,
0, None,
win32file.OPEN_EXISTING,
0, None)
read_fd = msvcrt.open_osfhandle(handle, os.O_RDONLY)
with open(read_fd, "r") as reader:
# now we have a file object with the readlines and other file api methods
lines = reader.readlines()
print(lines)
Some notes.
I've only tested this with python 3.4, but I believe you may be using python 2.x.
Python seems to get weird if you try to close both the file object and the pipe..., so I've only used the file object (by using the with block)
I've only created the file objects to read on one end and write on the other. You can of course make the file objects duplex by
Creating the file descriptors (read_fd and write_fd) with the os.O_RDWR flag
Creating the file objects in in "r+" mode rather than "r" or "w"
Going back to creating the pipe with the win32pipe.PIPE_ACCESS_DUPLEX flag
Going back to creating the file handle object with the win32file.GENERIC_READ | win32file.GENERIC_WRITE flags.

Python overwrite file containing current date and time

I am new to Python. I need to create a door.lock file that contains the current date and time. Also, I need to overwrite this file every x minutes with a new file containing the current date and time. I'm using this as a pseudo lock file to allow me to test on run of the software whether or not the software crashed and how long ago it crashed. My issue is I can't seem to overwrite the file. I've only failed at creating and/or appending the file. I created the following as a test:
from datetime import datetime, timedelta
ending = False
LOCK_FILENAME = "door.lock" # The lock file
LOCK_FILE_UPDATE = True
MINS_LOCK_FILE_UPDATE = 1 # the (x) time in minutes to write to lock file
NEXT_LOCK_FILE_UPDATE = datetime.now()
lock_file = open(LOCK_FILENAME, "w")
now = datetime.now()
NOW_STRING1 = str(now.strftime("%Y-%m-%d_%a_%H:%M"))
lock_file.write(NOW_STRING1)
print "First Now String"
print NOW_STRING1
# ==============================================================================
#Main Loop:
while ending is False:
# ==============================================================================
# Check if it is time to do a LOCK FILE time update
now = datetime.now()
NOW_STRING1 = str(now.strftime("%Y-%m-%d_%a_%H:%M"))
if LOCK_FILE_UPDATE: # if LOCK_FILE_UPDATE is set to True in DM settings
if NEXT_LOCK_FILE_UPDATE <= datetime.now():
lock_file.write(NOW_STRING1)
print NOW_STRING1
NEXT_LOCK_FILE_UPDATE = datetime.now() + timedelta(minutes=MINS_LOCK_FILE_UPDATE)
Will someone pinpoint my error(s) for me? TIA
When I cat the above file, door.lock, it is empty.

You need to push buffer to file. You can do it with a close() and re-open for next write.
lock_file.close()
...
lock_file = open(LOCK_FILENAME, "a")

If you are logging events you'd be better using a logger instead of a plain text file.
Solution from #MAC will work except it will append and seems that you don't want to do that so just open again with the 'w' option or yet better, use the 'w+' option so it can be truncated (which for what I get it is what you want to do) and read.
Also, consider your changes won't get written down until you close the file (having said that, consider open/close inside your loop instead).
lock_file = open(LOCK_FILENAME, "w+")
now = datetime.now()
NOW_STRING1 = str(now.strftime("%Y-%m-%d_%a_%H:%M"))
lock_file.write(NOW_STRING1)
# your loop and so on ...
lock_file.close()