We Keep Coding

Using stdout and stdin in a loop in python leading to errors

I am using stdout and stdin to communicate information between two python programs. tester.py should pass telemetry data into helper.py and helper.py should return some command to tester.py.
This seems to work when run without a loop, but when I put the code in tester.py inside a loop that updates the telemetry data, helper.py no longer seems able to pass back the correct command. The console print out is as follows:
b'\x00\x00\x00\x00\x01\x00\x00\x00'
0.0
b''
Traceback (most recent call last):
File "/Users/Advay/Documents/PyCharm/zip_sim/tester.py", line 44, in <module>
varr = COMMAND_STRUCT.unpack(cmd)
struct.error: unpack requires a buffer of 8 bytes
The tester.py:
import sys
import subprocess
import struct
TELEMETRY_STRUCT = struct.Struct(">fB3s")
COMMAND_STRUCT = struct.Struct(">fB3s")
helper = subprocess.Popen(['python3', 'helper.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
drop = 1
padding = bytes(3)
for i in range(5):
speed = i
helper.stdin.write(TELEMETRY_STRUCT.pack(speed, drop, padding))
helper.stdin.flush()
cmd = helper.stdout.read(COMMAND_STRUCT.size)
print(cmd)
varr = COMMAND_STRUCT.unpack(cmd)
print(varr[0])
and the helper.py:
import os
import random
import sys
import struct
TELEMETRY_STRUCT = struct.Struct(">fB3s")
COMMAND_STRUCT = struct.Struct(">fB3s")
telemetry = sys.stdin.buffer.read(TELEMETRY_STRUCT.size)
a = TELEMETRY_STRUCT.unpack(telemetry)
command = COMMAND_STRUCT.pack(a[0], 1, bytes(3))
sys.stdout.buffer.write(command)
sys.stdout.buffer.flush()
Any help would be appreciated a lot, I am at a complete loss as to why it. does not work in the loop.

You're trying to send multiple commands from tester.py to helper.py, but helper.py only reads a single command and then exits -- there is no loop that would allow it to continue receiving additional commands from tester.py.
When you run tester.py, the first loop iteration succeeds, but the subsequent iteration fails because the helper.stdout.read() returns an empty value (because the helper has exited).
You need to structure your helper.py so that it can receive multiple commands.
For example:
import os
import random
import sys
import struct
TELEMETRY_STRUCT = struct.Struct(">fB3s")
COMMAND_STRUCT = struct.Struct(">fB3s")
while True:
telemetry = sys.stdin.buffer.read(TELEMETRY_STRUCT.size)
if not telemetry:
break
a = TELEMETRY_STRUCT.unpack(telemetry)
command = COMMAND_STRUCT.pack(a[0], 1, bytes(3))
sys.stdout.buffer.write(command)
sys.stdout.buffer.flush()
With this change, running tester.py results in:
b'\x00\x00\x00\x00\x01\x00\x00\x00'
0.0
b'?\x80\x00\x00\x01\x00\x00\x00'
1.0
b'#\x00\x00\x00\x01\x00\x00\x00'
2.0
b'##\x00\x00\x01\x00\x00\x00'
3.0
b'#\x80\x00\x00\x01\x00\x00\x00'
4.0

How to access a docstring from a separate script?

Building a GUI for users to select Python scripts they want to run. Each script has its own docstring explaining inputs and outputs for the script. I want to display that information in the UI once they've highlighted the script, but not selected to run it, and I can't seem to get access to the docstrings from the base program.
ex.
test.py
"""this is a docstring"""
print('hello world')
program.py
index is test.py for this example, but is normally not known because it's whatever the user has selected in the GUI.
# index is test.py
def on_selected(self, index):
script_path = self.tree_view_model.filePath(index)
fparse = ast.parse(''.join(open(script_path)))
self.textBrowser_description.setPlainText(ast.get_docstring(fparse))

Let's the docstring you want to access belongs to the file, file.py.
You can get the docstring by doing the following:
import file
print(file.__doc__)
If you want to get the docstring before you import it then the you could read the file and extract the docstring. Here is an example:
import re
def get_docstring(file)
with open(file, "r") as f:
content = f.read() # read file
quote = content[0] # get type of quote
pattern = re.compile(rf"^{quote}{quote}{quote}[^{quote}]*{quote}{quote}{quote}") # create docstring pattern
return re.findall(pattern, content)[0][3:-3] # return docstring without quotes
print(get_docstring("file.py"))
Note: For this regex to work the docstring will need to be at the very top.

Here's how to get it via importlib. Most of the logic has been put in a function. Note that using importlib does import the script (which causes all its top-level statements to be executed), but the module itself is discarded when the function returns.
If this was the script docstring_test.py in the current directory that I wanted to get the docstring from:
""" this is a multiline
docstring.
"""
print('hello world')
Here's how to do it:
import importlib.util
def get_docstring(script_name, script_path):
spec = importlib.util.spec_from_file_location(script_name, script_path)
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
return foo.__doc__
if __name__ == '__main__':
print(get_docstring('docstring_test', "./docstring_test.py"))
Output:
hello world
this is a multiline
docstring.
Update:
Here's how to do it by letting the ast module in the standard library do the parsing which avoids both importing/executing the script as well as trying to parse it yourself with a regex.
This looks more-or-less equivalent to what's in your question, so it's unclear why what you have isn't working for you.
import ast
def get_docstring(script_path):
with open(script_path, 'r') as file:
tree = ast.parse(file.read())
return ast.get_docstring(tree, clean=False)
if __name__ == '__main__':
print(repr(get_docstring('./docstring_test.py')))
Output:
' this is a multiline\n docstring.\n'

How to fix delays/lag in python script

I have my python script that executes an mp3 when the current time matches the time specified in a text file. However everything works well but I notice a lag and delay of around 18 seconds before mplayer plays the mp3 file.
Is there anyway of making my python script better in order to get rid of the 18 seconds lag and make the mp3 file play instantaneously?
Here is my python script:
#!/usr/bin/python
# -*- coding: utf-8 -*-
# import libraries
import json
import urllib2
from bs4 import BeautifulSoup
import requests
import datetime
import playsound
import os
import subprocess
dateSTR = datetime.datetime.now().strftime('%H:%M')
f = open('/home/pi/test.txt','r')
messagetest = f.read()
newnametest = messagetest.replace("\n","")
f.close()
if (dateSTR) == (newnametest):
os.system("mplayer -ao alsa:device=bluealsa /home/pi/test.mp3")

Try starting mplayer in a subprocess before you actually need it as:
p = subprocess.Popen('mplayer -slave -idle -ao alsa:device=bluealsa', shell=True, stdin=subprocess.PIPE)
That should start up mplayer and have it waiting for when you need it. Then, when you want to play a file, do:
p.communicate(input=b'loadfile /home/pi/test.mp3\n')

I'd create a loop, something like:
from time import sleep
from datetime import datetime
...
done = []
while 1:
dateSTR = datetime.now().strftime('%H:%M')
if (dateSTR) == (newnametest) and not dateSTR in done:
done.append(dateSTR)
os.system("mplayer -ao alsa:device=bluealsa /home/pi/test.mp3")
sleep(1)

Python subprocess script failing

Have written the below script to delete files in a folder not matching the dates in the "keep" period. Eg. Delete all except files partly matching this name.
The command works from the shell but fails with the subprocess call.
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import subprocess
### Editable Variables
keepdays=7
location="/home/backups"
count=0
date_string=''
for count in range(0,keepdays):
if(date_string!=""):
date_string+="|"
keepdate = (datetime.now() - timedelta(days=count)).strftime("%Y%m%d")
date_string+="*\""+keepdate+"\"*"
full_cmd="/bin/rm "+location+"/!("+date_string+")"
subprocess.call([full_cmd], shell=True)
This is what the script returns:
#./test.py
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
/bin/sh: 1: Syntax error: "(" unexpected
Python version is Python 2.7.12

Just as #hjpotter said, subprocess will use /bin/sh as default shell, which doesn't support the kind of globbing you want to do. See official documentation. You can change that using the executable parameter to subprocess.call() with a more appropriate shell (/bin/bash or /bin/zsh for example): subprocess.call([full_cmd], executable="/bin/bash", shell=True)
BUT you can be a lot better served by Python itself, you don't need to call a subprocess to delete a file:
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import re
import os
import os.path
### Editable Variables
keepdays=7
location="/home/backups"
now = datetime.now()
keeppatterns = set((now - timedelta(days=count)).strftime("%Y%m%d") for count in range(0, keepdays))
for filename in os.listdir(location):
dates = set(re.findall(r"\d{8}", filename))
if not dates or dates.isdisjoint(keeppatterns):
abs_path = os.path.join(location, filename)
print("I am about to remove", abs_path)
# uncomment the line below when you are sure it won't delete any valuable file
#os.path.delete(abs_path)

Passing command line argument to another file imported in Python

I have a python file (html2text.py) which gives the desired result when i pass command line argument to it i.e., in the following way:
python html2text.py file.txt
where file.txt contains the source code of a web-site and the result is displayed on the console...
I want to use it in another file (let say a.py) and store the result (which was getting printed on the console) in a string.
For this I need to first import the file (html2text.py) in my file (a.py). Can anyone tell me how do I proceed further...?

Good way is to create some API in your html2text.py. For example:
# html2text.py
def parse(filename):
f = open(filename)
# do the stuff
return output_string
def main():
import sys
print parse(sys.argv[1])
if __name__ == '__main__':
main()
Then you will be able to use it in your a.py:
import html2text # main() will not run
import sys
output = html2text.parse(sys.argv[1])

I think the best way is the reorganize a little your html2text.py file. Append the line like this to your file:
def main():
message = sys.stdin.readlines()
a = your_def(message)
if __name__ == '__main__': main()
Now you're sure, that when invoking the file from command line, everything will go fine. Moreover, if you have everything kept in functions and classes, you can now in your a.py
import html2text
and work on it already in a.py.