We Keep Coding

PyVisa Data Extraction Issues with Keysight B1500

I have a similar question as this one but the solution there did not apply to my problem. I can connect and send commands to my Keysight B1500 mainframe, via pyvisa/GPIB. The B1500 is connected via Keysight's IO tool "Connection Expert"
rman = pyvisa.ResourceManager()
keyS = rman.open_resource('GPIB0::18::INSTR')
keyS.timeout = 20000 # time in ms
keyS.chunk_size = 8204800 # 102400 is 100 kB
keyS.write('*rst; status:preset; *cls')
print('variable keyS is being assigned to ',keyS.query('*IDN?'))
Using this pyvisa object I can query without issues (*IDN? above provides the expected output), and I have also run and extracted data from a different type of IV curve on the same tool.
However, when I try to run a pulsed voltage sweep (change voltage of pulses as function of time and measure current) I do not get the measured data out from the tool. I can hook the output lead from the B1500 to an oscilloscope and can see that my setup has worked and the tool is behaving as expected, right up until I try to extract sweep data.
Again, I can run a standard non-pulsed sweep on the tool and the data extraction works fine using [pyvisaobject].read_raw() - so something is different with the way I'm pulsing the voltage.
What I'm looking for is a way to interrogate the connection in cases where the data transfer is unsuccessful.
Here, in no particular order, are the ways I've tried to extract data. These methods are suggested in this link:
keyS.query_ascii_values('CURV?')
or
keyS.read_ascii_values()
or
keyS.query_binary_values('CURV?')
or
keyS.read_binary_values()
This link from the vendor does cover the extraction of data, but also doesn't yield data in the read statement in the case of pulsed voltage sweep:
myFieldFox.write("TRACE:DATA?")
ff_SA_Trace_Data = myFieldFox.read()
Also tried (based on tab autocompletion on iPython):
read_raw() # This is the one that works with non-pulsed sweep
and
read_bytes(nbytes)
The suggestion from #Paul-Cornelius is a good one, I had to include an *OPC? to get the previous data transfer to work as well. So right before I attempt the data transfer, I send these lines:
rep = keyS.query('NUB?')
keyS.query('*OPC?')
print(rep,'AAAAAAAAAAAAAAAAAAAAA') # this line prints!
mretholder = keyS.read_raw() # system hangs here!
In all the cases the end result is the same - I get a timeout error:
pyvisa.errors.VisaIOError: VI_ERROR_TMO (-1073807339): Timeout expired before operation completed.
The tracebacks for all of these show that they are all using the same basic framework from:
chunk, status = self.visalib.read(self.session, size)
Hoping someone has seen this before, or at least has some ideas on how to troubleshoot. Thanks in advance!

I don't have access to that instrument, but to read a waveform from a Tektronics oscilloscope I had to do the following (pyvisa module):
On obtaining the Resource, I do this:
resource.timeout = 10000
In some cases, I have to wait for a command to complete before proceeding like this:
resource.query("*opc?")
To transfer a waveform from the scope to the PC I do this:
ascii_waveform = resource.query("wavf?")
The "wavf?" query is specifically for this instrument, but the "*opc?" query is generic ("wait for operation complete"), and so is the method for setting the timeout parameter.
There is a lot of information in the user's guide (on the same site you have linked to) about reading data from various devices. I have used the visa library a few times on different devices, and it always requires some fiddling to get it to work.
It looks like you don't have any time delays in your program. In my experience they are almost always necessary, somehow, either by using the resource.timeout feature, the *opc? query, or both.

Writing fast serial data to a file (csv or txt)

Is there a way to capture and write very fast serial data to a file?
I'm using a 32kSPS external ADC and a baud rate of 2000000 while printing in the following format: adc_value (32bits) \t millis()
This results in ~15 prints every 1 ms. Unfortunately every single soulution I have tried fails to capture and store real time data to a file. This includes: Processing sketches, TeraTerm, Serial Port Monitor, puTTY and some Python scripts. All of them are unable to log the data in real time.
Arduino Serial Monitor on the other hand is able to display real time serial data, but it's unable to log it in a file, as it lacks this function.
Here's a printscreen of the serial monitor in Arduino with the incoming data:

One problematic thing is probably that you try to do a write each time you receive a new record. That will waste a lot of time writing data.
Instead try to collect the data into buffers, and as a buffer is about to overflow write the whole buffer in a single and as low-level as possible write call.
And to not stop the receiving of the data to much, you could use threads and double-buffering: Receive data in one thread, write to a buffer. When the buffer is about to overflow signal a second thread and switch to a second buffer. The other thread takes the full buffer and writes it to disk, and waits for the next buffer to become full.

After trying more than 10 possible solutions for this problem including dedicated serial capture software, python scripts, Matlab scripts, and some C projects alternatives, the only one that kinda worked for me proved to be MegunoLink Pro.
It does not achieve the full 32kSPS potential of the ADC, rather around 12-15kSPS, but it is still much better than anything I've tried.
Not achieving the full 32kSPS might also be limited by the Serial.print() method that I'm using for printing values to the serial console. By the way, the platform I've been using is ESP32.
Later edit: don't forget to edit MegunoLinkPro.exe.config file in the MegunoLink Pro install directory in order to add further baud rates, like 1000000 or 2000000. By default it is limited to 500000.

how do I monitor data from serial port and have certain data act as a flag in Python

I am working on a project for work and I am stuck on a part where I need to monitor a serial line and listen for certain words using python
so the setup is that we have a automated RAM testing machine that tests RAM one module at a time and interacts with software that came with the machine via serial. The software that came with the RAM tester is for monitoring/configuring the testing process, it also displays all of the information from the SPD chip from each module. while the RAM tester was running I ran a serial port monitoring program and I was able to see the same information that it displays in the software. The data I'm interested in is the speed of the RAM and the pass/fail result, both of which I was able to find in the data I monitored coming over the serial line. There are only 5 different speeds of RAM that we test, so I was hoping to have python monitor the serial line and wait for the speed of the RAM and the pass/fail results to come across. once python detects the speed of the RAM, and if it passes, I will have python write to an Arduino, and the Arduino will control a conveyor belt that will sort the ram by speed.
my idea is to have a variable for each of the RAM speeds and set the variables to 0. when python detects the RAM speed from the serial line it will set the corresponding variable to 1. then when the test is over the results, either pass or fail, will come over the serial line. this is where I am going to try to use a if statement. I imagine it would look something like this:
if PC-6400 == 1 and ser.read() == pass
ser.write(PC-6400) #serial write to the arduino
I know the use of the ser.read() == pass is incorrect and that's where I'm stuck. I do not know how to use a ser.read() function to look for certain words. I need it to look for the ram speed (in this case its PC-6400) and the word pass but I have not been successful in getting it to find either. I am currently suck in is getting python to detect the RAM speed so it can change the value of the variable. would it be something close to this?
if ser.read() == PC-6400
PC-6400 = 1
This whole thing is a bit difficult for me to explain and I hope it all makes sense, I thank you in advance if anyone can give me some advice on how to get this going. I am pretty new to python and this is the most adventurous project I have worked on using python so far.

I'm still a bit confused, but here's a very basic example to hopefully get you started.
This will read from a serial port. It will attempt to read 512 bytes (which just means 512 characters from a string). If 512 bytes aren't available then it will wait forever, so make sure you set a timeout when you made the serial connection.
return_str = ser.read(size = 512)
You can also see how many bytes are waiting to be read:
print "num bytes available = ", ser.inWaiting()
Once you have a string, you can check words within the string:
if "PASS" in return_str:
print "the module passed!"
if "PC-6400" in return_str:
print "module type is PC-6400"
To do something similar, but with variables:
# the name pass is reserved
pass_flag = "PASS"
PC6400 = 0
if pass_flag in return_str and "PC-6400" in return_str:
PC6400 = 1
Keep in mind that you it is possible to read part of a line if you are too quick. You can add delays by using timeouts or the time.sleep() function. You might also find you need to wait for a couple of seconds after initiating the connection as the arduino resets when you connect. This will give it time to reset before you try and read/write.

Python socket wait

I was wondering if there is a way I can tell python to wait until it gets a response from a server to continue running.
I am writing a turn based game. I make the first move and it sends the move to the server and then the server to the other computer. The problem comes here. As it is no longer my turn I want my game to wait until it gets a response from the server (wait until the other player makes a move). But my line:
data=self.sock.recv(1024)
hangs because (I think) it's no getting something immediately. So I want know how can I make it wait for something to happen and then keep going.
Thanks in advance.

The socket programming howto is relevant to this question, specifically this part:
Now we come to the major stumbling block of sockets - send and recv operate on the
network buffers. They do not necessarily handle all the bytes you hand them (or expect
from them), because their major focus is handling the network buffers. In general, they
return when the associated network buffers have been filled (send) or emptied (recv).
They then tell you how many bytes they handled. It is your responsibility to call them
again until your message has been completely dealt with.
...
One complication to be aware of: if your conversational protocol allows multiple
messages to be sent back to back (without some kind of reply), and you pass recv an
arbitrary chunk size, you may end up reading the start of a following message. You’ll
need to put that aside >and hold onto it, until it’s needed.
Prefixing the message with it’s length (say, as 5 numeric characters) gets more complex,
because (believe it or not), you may not get all 5 characters in one recv. In playing
around, you’ll get away with it; but in high network loads, your code will very quickly
break unless you use two recv loops - the first to determine the length, the second to
get the data part of the message. Nasty. This is also when you’ll discover that send
does not always manage to get rid of everything in one pass. And despite having read
this, you will eventually get bit by it!
The main takeaways from this are:
you'll need to establish either a FIXED message size, OR you'll need to send the the size of the message at the beginning of the message
when calling socket.recv, pass number of bytes you actually want (and I'm guessing you don't actually want 1024 bytes). Then use LOOPs because you are not guaranteed to get all you want in a single call.

That line, sock.recv(1024), blocks until 1024 bytes have been received or the OS detects a socket error. You need some way to know the message size -- this is why HTTP messages include the Content-Length.
You can set a timeout with socket.settimeout to abort reading entirely if the expected number of bytes doesn't arrive before a timeout.
You can also explore Python's non-blocking sockets using setblocking(0).

Python/Scapy Performance on embedded device using Wi-Fi

I am using Scapy to capture Wi-Fi client request frames. I am only interested in the MAC and requested SSID address of the clients. I do something like the following.
sniff(iface="mon0", prn=Handler)
def Handler(pkt):
if pkt.hasLayer(Dot11):
if pkt.type == 0 and pkt.subtype == 4:
print pkt.addr2 + " " + pkt.info
My issue is that I am doing this on a embedded device with limited processing power. When I run my script, my processor utilization rises to nearly 100%. I assume this is because of the sheer volume of frames that Scapy is sniffing and passing to my Python code. I also assume that if I could employ the right filter in my sniff command, I could eliminate many of the frames that are not being used and thus reduce the processor load.
Is there a filter statement that could be used to do this?

With Scapy it is possible to sniff with an BPF Filter applied to the capture. This will filter out packets at a much lower level then your Handler function is doing, and should significantly improve performance.
There is a simple example in the Scapy documentation.
# Could also add a subtype to this.
sniff(iface="mon0", prn=Handler, filter="type mgt")
Filter pieced together from here specifically.
Unfortunately I can't test this right now, but this information should provide you with a stepping stone to your ultimate solution, or someone else to post exactly what you need. I believe you will also need to set the interface to monitor mode.
You may also find this question of interest - Accessing 802.11 Wireless Management Frames from Python.

Scapy is extremely slow due to the way it decodes the data. You may
Use a BPF-filter on the input to only get the frames you are looking for before handing them to scapy. See this module for example. The module uses libpcap to get the data from the air or from a file first and passes it through a dynamically updated BPF-filter to keep unwanted traffic out
Write your own parser for wifi in c (which is not too hard, given the limited amount of information you need, there are things like prismhead though)
Use tshark from wireshark as a subprocess and collect data from there
I highly recommend the third approach although wiresharek comes with a >120mb library that your embedded device might not be able to handle.