We Keep Coding

Python psutil how to find a child process

Given PyQt5 on Linux, I have an application that starts a terminal emulator (rxvt) and runs a command (gaurdian) which runs yet another program (goo). Like this
medi#medi:~> pstree -p 4610
rxvt(4610)─┬─gaurdian(4612)───goo(4613)
└─rxvt(4611)
I am trying to find pid of "goo". So I proceed with
gooPID = 12 # some random value to show my point
self.process.start(cmd, cmdOpts)
rxvtPID = self.process.processId()
try:
for c in psutil.Process(rxvtPID).children(True):
print("pid=%d name=%s" % (c.pid, c.name()))
if c.name() == 'gaurdian':
gooPID = c.pid
except (psutil.ZombieProcess, psutil.AccessDenied, psutil.NoSuchProcess) as err:
print(err)
print("gooPID=%d " % gooPID )
The trace log is showing:
rxvtPID=4610 name=rxvt
gooPID=12
which suggests that the initial value of gooPID was not changed. Also seems like traversal of children is not happening (ie I am not seeing children of children, etc).
Am I doing this right ?

I managed to solve this by inserting a sleep(1) before psutil begins to traverse the /proc filesystem. That is
209 # ----------------------- getPidByName() ----------------------
210 def getPidByName(self, name):
211 rxvtProc = psutil.Process(self.process.processId() )
212 time.sleep(1) # else /proc is not ready for read
213 pid = None
214 try:
215 for c in rxvtProc.children(True):
216 # assumption: gaurdian has only one child
217 if c.name() == name:
218 return psutil.Process(c.pid).children()[0].pid
219 except psutil.Error as err:
220 print(err)
221 return pid

Unable to export in parallel from Exasol using pyexasol

I'm attempting to fetch data from Exasol using PyExasol, in parallel. I'm following the example here - https://github.com/badoo/pyexasol/blob/master/examples/14_parallel_export.py
My code looks like this :
import multiprocessing
import pyexasol
import pyexasol.callback as cb
class ExportProc(multiprocessing.Process):
def __init__(self, node):
self.node = node
self.read_pipe, self.write_pipe = multiprocessing.Pipe(False)
super().__init__()
def start(self):
super().start()
self.write_pipe.close()
def get_proxy(self):
return self.read_pipe.recv()
def run(self):
self.read_pipe.close()
http = pyexasol.http_transport(self.node['host'], self.node['port'], pyexasol.HTTP_EXPORT)
self.write_pipe.send(http.get_proxy())
self.write_pipe.close()
pd1 = http.export_to_callback(cb.export_to_pandas, None)
print(f"{self.node['idx']}:{len(pd)}")
EXASOL_HOST = "<IP-ADDRESS>:8563"
EXASOL_USERID = "username"
EXASOL_PASSWORD = "password"
c = pyexasol.connect(dsn=EXASOL_HOST, user=EXASOL_USERID, password=EXASOL_PASSWORD, compression=True)
nodes = c.get_nodes(10)
pool = list()
proxy_list = list()
for n in nodes:
proc = ExportProc(n)
proc.start()
proxy_list.append(proc.get_proxy())
pool.append(proc)
c.export_parallel(proxy_list, "SELECT * FROM SOME_SCHEMA.SOME_TABLE", export_params={'with_column_names': True})
stmt = c.last_statement()
r = stmt.fetchall()
At the last statement, I'm getting the following error and unable to fetch any results.
---------------------------------------------------------------------------
ExaRuntimeError Traceback (most recent call last)
<command-911615> in <module>
----> 1 r = stmt.fetchall()
/local_disk0/pythonVirtualEnvDirs/virtualEnv-01515a25-967f-4b98-aa10-6ac03c978ce2/lib/python3.7/site-packages/pyexasol/statement.py in fetchall(self)
85
86 def fetchall(self):
---> 87 return [row for row in self]
88
89 def fetchcol(self):
/local_disk0/pythonVirtualEnvDirs/virtualEnv-01515a25-967f-4b98-aa10-6ac03c978ce2/lib/python3.7/site-packages/pyexasol/statement.py in <listcomp>(.0)
85
86 def fetchall(self):
---> 87 return [row for row in self]
88
89 def fetchcol(self):
/local_disk0/pythonVirtualEnvDirs/virtualEnv-01515a25-967f-4b98-aa10-6ac03c978ce2/lib/python3.7/site-packages/pyexasol/statement.py in __next__(self)
53 if self.pos_total >= self.num_rows_total:
54 if self.result_type != 'resultSet':
---> 55 raise ExaRuntimeError(self.connection, 'Attempt to fetch from statement without result set')
56
57 raise StopIteration
ExaRuntimeError:
(
message => Attempt to fetch from statement without result set
dsn => <IP-ADDRESS>:8563
user => username
schema =>
)
It seems that the type of the returned statement is not 'resultSet' but 'rowCount'. Any help on what I'm doing wrong or why the type of statement is ''rowCount' ?

PyEXASOL creator is here. Please not in case of parallel HTTP transport you have to process data chunks inside child processes. Your data set is available in pd1 DataFrame.
You should not be calling .fetchall() in the main process in case of parallel processing.
I suggest to check the complete examples, especially example 14 (parallel export).
Hope it helps!

Pymodbus : Wrong byte count in response

We are requesting 14 responses from an RS485 device and sometimes the reply that we got doesn't have the 9 bytes that it is set-up. That is because it's replying sometimes in 3 arguments.
Normal:
CALL-> 01 04 00 00 00 02 71 CB
RESPONSE-> 01 04 04 43 59 E6 66 F4 59
Error:
CALL-> 01 04 00 00 00 02 71 CB
RESPONSE -> 01 04 04 43
59 CC CD AA 86
When the error happens i get this msg from pymodbus:
DEBUG:pymodbus.transaction: Incomplete message received, Expected 9 bytes Recieved 4 bytes !!!!
DEBUG:pymodbus.transaction:Changing transaction state from 'WAITING FOR REPLY' to 'PROCESSING REPLY'
DEBUG:pymodbus.transaction:RECV: 0x1 0x4 0x4 0x3e
DEBUG:pymodbus.framer.rtu_framer:Frame check failed, ignoring!!
DEBUG:pymodbus.framer.rtu_framer:Resetting frame - Current Frame in buffer - 0x1 0x4 0x4 0x3e
DEBUG:pymodbus.transaction:Getting transaction 1
DEBUG:pymodbus.transaction:Changing transaction state from 'PROCESSING REPLY' to 'TRANSACTION_COMPLETE'
i've tried putting a sleep into the for so it dosn't colapse the device with calls but i get them either way. i've also read https://wingpath.co.uk/docs/modtest/troubleshoot.html
and they say this:
"Wrong byte count in response: XXX when expecting XXX"
The byte count in the response sent by the slave is not what was expected for the count that ModTest sent in the request.
Turn on tracing to get more information.
Check that your slave is functioning correctly.
If you want ModTest to accept the response even though it is incorrect, you could deselect Strict Checking.
But i don't know how to active tracing on PYMODBUS, the function is correct and the other one is for a lib that i am not using i think
THE CODE LOOKS LIKE THIS
from __future__ import division
import pymodbus
import serial
from pymodbus.pdu import ModbusRequest
from pymodbus.client.sync import ModbusSerialClient as ModbusClient #initialize a serial RTU client instance
from pymodbus.transaction import ModbusRtuFramer
from time import sleep
from pymodbus.constants import Endian # Nodig voor 32-bit float getallen (2 registers / 4 bytes)
from pymodbus.payload import BinaryPayloadDecoder # Nodig voor 32-bit float getallen (2 registers / 4 bytes)
from pymodbus.payload import BinaryPayloadBuilder # Nodig om 32-bit floats te schrijven naar register
import logging
logging.basicConfig()
log = logging.getLogger()
log.setLevel(logging.DEBUG)
#
method = "rtu"
port = "COM1"
baudrate = 2400
stopbits = 1
bytesize = 8
parity = "N"
timeout = 10 # I SET THIS TO 10 MAYBE IT WOULD HELP BUT DIDN'T
retries = 5 # SAME THING WITH THIS ONE
#
try:
client = ModbusClient(method = method, port = port, stopbits = stopbits, bytesize = bytesize, parity = parity, baudrate = baudrate, timeout = timeout, retries = retries)
connection = client.connect()
print (connection)
except:
print ("Modbus connectie error")
#
def 420 (y):
variables = [0,6,12,18,24,30,36,70,72,74,76,78,342,344]
labels = ["Voltage","Corriente","Potencia_Activa","Potencia_Aparente","Potencia_Reactiva","Factor_Potencia","Angulo_Fase","Frecuencia","Potencial_Activa_Consumida","Potencia_Activa_Inyectada","Potencia_Reactiva_Consumida","Potencia_Reactiva_Inyectada","Energia_Activa_Total","Energia_Reactiva_Total"]
unidades = ["V","A","W","VA","VAr","%","Grados","HZ","kWh","kWh","kVArh","kVArh","kWh","kVArh"]
LISTA = []
h = 0
hh = 0
for xx in variables:
try:
data = client.read_input_registers(xx, 2, unit=1)
decoder = BinaryPayloadDecoder.fromRegisters(data.registers, Endian.Big)
eastron = round(decoder.decode_32bit_float(), 3)
weaito = str(labels[h]) + " = " + str(eastron) + " " + str(unidades[hh])
LISTA.append(weaito)
h = h + 1
hh = hh + 1
sleep(0.5)
except:
print ("PICO")
sleep(1)
print(LISTA)
I Would love a way around the problem, maybe just consulting again until i get the right answer. I am not good with the try and except ones maybe there is the answer.

You seem to be experiencing a known issue with interchar spacing.
There is an easy workaround though. First, make sure you're on pymodbus version 2.2.0 (you can do that opening a command line terminal on Windows and typing pip list if you have the path setup correctly, otherwise you have to move to your scripts Python folder where pip.exe is stored).
Then change your code to add the strict argument declared to False:
....
client = ModbusClient(method = method, port = port, stopbits = stopbits, bytesize = bytesize, parity = parity, baudrate = baudrate, timeout = timeout, retries = retries)
client.strict = False #Use Modbus interchar spacing as timeout to prevent missing data for low baudrates
connection = client.connect()
...
This will define the character spacing according to the Modbus spec, to 1.5 times the bit time at the selected baudrate, instead of using the default from socket.interCharTimeout:
self._t0 = float((1 + 8 + 2)) / self.baudrate
self.inter_char_timeout = 1.5 * self._t0
If you can fix your issue with this solution, you should be able to reduce the overhead on your device reading the 28 registers you want in one go.
If your issue is not solved, I think you might have a hardware issue. I would advise you to put your code aside for a while and try reading registers with QModMaster or something similar to make sure your hardware is performing as intended. (you might be getting noise or grounding issues that cut your frames short, if you want some pointers on that front please edit your question to include more details on your hardware, i.e.: kind of devices and how you're connecting them).

Parsing a large space separated file into sqlite

I am trying to parse a large space separated file (3 GB and higher) into a sqlite database for other processing. The file currently has around 20+ million lines of data. I have tried multithreading this, but for some reason, it stops with around 1500 lines and does not proceed. I don’t know if I am doing anything wrong. Can someone please point me in the right direction?
The insertion is working fine with one process, but is is way too slow (of course!!!). It has been running for over seven hours and it is not even past the first set of strings. The DB file is still 25 MB in size and not even close to the number of records it has to contain.
Please guide me towards speeding this up. I have one more huge file to go (more than 5 GB) and this could take days.
Here’s my code:
1 import time
2 import queue
3 import threading
4 import sys
5 import sqlite3 as sql
6
7 record_count = 0
8 DB_INSERT_LOCK = threading.Lock()
9
10 def process_data(in_queue):
11 global record_count
12 try:
13 mp_db_connection = sql.connect("sequences_test.sqlite")
14 sql_handler = mp_db_connection.cursor()
15 except sql.Error as error:
16 print("Error while creating database connection: ", error.args[0])
17 while True:
18 line = in_queue.get()
19 # print(line)
20 if (line[0] == '#'):
21 pass
22 else:
23 (sequence_id, field1, sequence_type, sequence_count, field2, field3,
24 field4, field5, field6, sequence_info, kmer_length, field7, field8,
25 field9, field10, field11, field12, field13, field14, field15) =
line.expandtabs(1).split(" ")
26
27 info = (field7 + " " + field8 + " " + field9 + " " + field10 + " " +
28 field11 + " " + field12 + " " + field13 + " " + field14 + " "
29 + field15)
30
31 insert_tuple = (None, sequence_id, field1, sequence_type, sequence_count,
32 field2, field3, field4, field5, field6, sequence_info,
33 kmer_length, info)
34 try:
35 with DB_INSERT_LOCK:
36 sql_string = 'insert into sequence_info \
37 values (?,?,?,?,?,?,?,?,?,?,?,?,?)'
38 sql_handler.execute(sql_string, insert_tuple)
39 record_count = record_count + 1
40 mp_db_connection.commit()
41 except sql.Error as error:
42 print("Error while inserting service into database: ", error.args[0])
43 in_queue.task_done()
44
45 if __name__ == "__main__":
46 try:
47 print("Trying to open database connection")
48 mp_db_connection = sql.connect("sequences_test.sqlite")
49 sql_handler = mp_db_connection.cursor()
50 sql_string = '''SELECT name FROM sqlite_master \
51 WHERE type='table' AND name='sequence_info' '''
52 sql_handler.execute(sql_string)
53 result = sql_handler.fetchone()
54 if(not result):
55 print("Creating table")
56 sql_handler.execute('''create table sequence_info
57 (row_id integer primary key, sequence_id real, field1
58 integer, sequence_type text, sequence_count real,
59 field2 integer, field3 text,
60 field4 text, field5 integer, field6 integer,
61 sequence_info text, kmer_length text, info text)''')
62 mp_db_connection.commit()
63 else:
64 pass
65 mp_db_connection.close()
66 except sql.Error as error:
67 print("An error has occured.: ", error.args[0])
68
69 thread_count = 4
70 work = queue.Queue()
71
72 for i in range(thread_count):
73 thread = threading.Thread(target=process_data, args=(work,))
74 thread.daemon = True
75 thread.start()
76
77 with open("out.txt", mode='r') as inFile:
78 for line in inFile:
79 work.put(line)
80
81 work.join()
82
83 print("Final Record Count: ", record_count)
The reason I have a lock is that with sqlite, I don’t currently have a way to batch commit my files into the DB and hence I have to make sure that every time a thread inserts a record, state of the DB is committed.
I know I am losing some processing time with the expandtabs call in the thick of things, but it is a little difficult to post process the file I am receiving to do a simple split on it. I will continue trying to do that so that the workload is reduced, but I need the multithreading at least to work.
EDIT:
I moved the expandtabs and split part outside the processing. So I process the line and insert in into the queue as a tuple so that the threads can pick it up and directly insert it into the DB. I was hoping to save quite a bit of time with this, but now I am running into problems with sqlite. It says could not insert into db because it is locked. I am thinking it is more of a thread sync issue with the locking part since I have an exclusive lock on the critical section below. Could someone please elaborate on how to resolve this?

I wouldn't expect multithreading to be of much use there. You should maybe write a generator function that processes the file into tuples, which you then insert with executemany

In addition to previous responses, try:
Use executemany from Connetion object
Use Pypy

multihtreading will not help you
The first thing you have to do is not commit each record according to
http://sqlite.org/speed.html it's a factor 250 in speed.
To not lose all your work if you interrupt just commit every 10000 or 100000 records

How to Capture the RFID Card's UID by just flashing the Card over the reader using Python2.7?

I have a RFID project, and wants the system to detect the card on the card reader as it is in read range and capture the UID and continue the process. As of now I have placed a button called ScanCard, in it I have placed the card read functionality, which would return the UID of the Card. I am using just two type of ATR. Want to get rid of the Scan Card button and want to automate the scanning function. I am using Python 2.7 and HID Omnikey Card Reader on Windows 7
atr = "3B 8F 80 01 80 4F 0C A0 00 00 03 06 0A 00 18 00 00 00 00 7A"
cardtype = ATRCardType( toBytes( "%s" %(atr) ))
cardrequest = CardRequest( timeout=1, cardType=cardtype )
cardservice = cardrequest.waitforcard()
cardservice.connection.connect()
SELECT = [0xFF, 0xCA, 0x00, 0x00, 0x00]
apdu = SELECT
print 'sending ' + toHexString(apdu)
response, sw1, sw2 = cardservice.connection.transmit( apdu )
print 'response: ', response, ' status words: ', "%x %x" % (sw1, sw2)
tagid = toHexString(response).replace(' ','')
print "tagid ",tagid
id = tagid
print" UID is",id
The above code is what I am following now. I need to keep the wait for card unlimited, what could be a optimum way to do it?

Maybe try the official pyscard documentation, such as the part on monitoring, which I have linked to.