Alright, I've found something. Not sure how to tackle it. I've seen that this is a common error that comes up in google. The error seems to have something to do with the environment variables or something. Not sure how to handle this:
This is the code and it's the part where subprocess is called that leads to the error:
#!/usr/bin/python
import subprocess
import re
import sys
import time
import datetime
import gspread
# ===========================================================================
# Google Account Details
# ===========================================================================
# Account details for google docs
email = 'my_email#gmail.com'
password = 'my_password'
spreadsheet = 'my_spreadsheet'
# ===========================================================================
# Example Code
# ===========================================================================
# Login with your Google account
try:
gc = gspread.login(email, password)
except:
print "Unable to log in. Check your email address/password"
sys.exit()
# Open a worksheet from your spreadsheet using the filename
try:
worksheet = gc.open(spreadsheet).sheet1
# Alternatively, open a spreadsheet using the spreadsheet's key
# worksheet = gc.open_by_key('0BmgG6nO_6dprdS1MN3d3MkdPa142WFRrdnRRUWl1UFE')
except:
print "Unable to open the spreadsheet. Check your filename: %s" % spreadsheet
sys.exit()
# Continuously append data
while(True):
# Run the DHT program to get the humidity and temperature readings!
output = subprocess.check_output(["./Adafruit_DHT", "2302", "17"]);
print output
matches = re.search("Temp =\s+([0-9.]+)", output)
if (not matches):
time.sleep(3)
continue
temp1 = float(matches.group(1))
temp = temp1*9/5+32 # added the extra step to converto to fahrenheit
# search for humidity printout
matches = re.search("Hum =\s+([0-9.]+)", output)
if (not matches):
time.sleep(3)
continue
humidity = float(matches.group(1))
print "Temperature: %.1f F" % temp
print "Humidity: %.1f %%" % humidity
# Append the data in the spreadsheet, including a timestamp
try:
values = [datetime.datetime.now(), temp, humidity]
worksheet.append_row(values)
except:
print "Unable to append data. Check your connection?"
sys.exit()
# Wait 30 seconds before continuing or just exit
print "Wrote a row to %s" % spreadsheet
# time.sleep(60)
sys.exit()
that's basically it. It works fine using 'sudo python script.py' as long as the Adafruit_DHT program is in the same directory.
Here's the error I get:
Traceback (most recent call last):
File "/home/pi/Adafruit/dht/Ada_temp.py", line 44, in <module>
output = subprocess.check_output(["./home/pi/Adafruit/dht/Adafruit_DHT", "2302", "17"]);
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I've tried adding the full path of the c program (Adafruit_DHT), to no avail...
Locate the problem.
Does the script run at all?
Do something trivial in the first line of the script to see that it actually gets executed from cron (e.g.: write to a file in /tmp).
Once you managed to run it, look for other problems. Cron can be set up to send the a mail with the output of the script. One obvious thing I see is: ./Adafruit_DHT, relative path is used, that's a bad sign, the cron script is probably not executed in the directory you think it does. Fix it (= use absolute path).
Try:
output = subprocess.check_output(["PATH_TO_Adafruit_DHT/Adafruit_DHT", "2302", "17"]);
Oh and change your cron line to:
/usr/bin/python /home/pi/myscript.py
Related
Screenshot I am trying to write a script that runs a .exe file using a silent installer with python. My main problem is that when the installer finishes it request a user input - see screenshot attached and the script will not finish until the user hit any key to start the services
Therefore, I wrote a script that has two functions running asynchronous
As expected I am having errors
My expectations: Function click_window will wait for function run_installer to finish and then it will click enter in the cmd.exe window - I will add a screenshot
What is happening: The Script runs as expected but when it gets to os.system = (file + '/SILENT') base on the debugger - it dies out with an error message
import asyncio
import subprocess
import time
import pyautogui
import pywinauto
from pyautogui import getWindowsWithTitle
import pygetwindow
async def click_window():
# Wait for everything to load in, windows, apps etc.
# You may need to adjust this timing for your system.
time.sleep(10)
# We are waiting for the window to open up and then click ok
app = pywinauto.Application()
# Tell Pyautogui what the windows name is.
Win = "C:\\WINDOWS\\System32\\cmd.exe"
# wait for the window to appear
app = pywinauto.Application()
app.WindowSpecification.wait(Win)
app.window_(title=Win).SetFocus()
app.window_(title=Win).TypeKeys("Enter")
# keyboard shortcut R to hit 'Run Cleaner' button,
pyautogui.press('Enter')
# Wait 15 seconds to give time for the cleaning operation.
# You may need to adjust.
time.sleep(15)
# Move mouse to exit button and click.
# pyautogui.moveTo(1905, 10, 1)
pyautogui.click()
print("We did it")
async def run_installer():
import fnmatch
import glob
import os
import shutil
import time
import urllib.request
import zipfile
# Download the installer
url = 'http://build.****.com:8080/job/CI_build_***_grs_master_windows_mysql/lastBuild/artifact/com.****.sdm.****/target/release/*zip*/release.zip'
file_name = 'release.zip'
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
time.sleep(120)
print("Download is done")
file_name = "release.zip"
with zipfile.ZipFile(file_name, 'r') as zip:
# printing all the contents of the zip file
zip.printdir()
time.sleep(60)
# extracting all the files
print('Extracting all the files now...')
zip.extractall()
print('I am done extracting Done!')
print("Path at terminal when executing this file")
print(os.getcwd() + "\n")
os.chdir("C:\\Users\\****\\PycharmProjects\\Phyton-Project\\release")
print("Path at terminal when executing this file again")
print(os.getcwd() + "\n")
# Checking installer directory
for file in glob.glob("*.exe"):
print(file + " Here is the file found")
# file variable contains installer
time.sleep(120)
# Adding permissions to installer to run/execute
os.chmod("C:\\Users\\****\\PycharmProjects\\Phyton-Project\\release", 0o777) # read/write by everyone
for file in os.listdir('.'):
if fnmatch.fnmatch(file, '*.exe'):
print("here is the file" + file)
# file variable is the exe installer
time.sleep(120)
print("Installation starts")
os.system = (file + '/SILENT')
# time.sleep(360)
start = time.time()
loop = asyncio.get_event_loop()
tasks = [
asyncio.gather(run_installer()),
asyncio.gather(click_window()),
]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
end = time.time()
print("Total time: {}".format(end - start))
Expected Results:
Function run installer finish and the window opens up to start up the services and then the click_window function starts up and clicks on the window and after 4 clicks for every service the script finish
Actual Results:
Function run installer runs but it dies at os.system = (file + '/SILENT') because the installer never installs and then it throws the following error message:
Connected to pydev debugger (build 192.5728.105)
I am done waiting
File Name Modified Size
release/pa****_**_5.4.3_20190903_win_x64_db.exe 2019-09-03 22:29:42 757267728
Extracting all the files now...
Done!
Path at terminal when executing this file
C:\Users\j****\PycharmProjects\Phyton-Project
Path at terminal when executing this file again
C:\Users\j*****\PycharmProjects\Phyton-Project\release
pa***_***_5.4.3_20190903_win_x64_db.exe Here is the file found
here is the fileparasoft_dtp_5.4.3_20190903_win_x64_db.exe
Installation starts
Total time: 485.42616963386536
_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=AppNotConnected('Please use start or connect before trying anything else')>
Traceback (most recent call last):
File "C:/Users/****/PycharmProjects/Phyton-Project/DTP-Installation.py", line 27, in click_window
app.WindowSpecification.wait(Win)
File "C:\Users\****\PycharmProjects\Phyton-Project\venv\lib\site-packages\pywinauto\application.py", line 1234, in __getattribute__
return self[attr_name]
File "C:\Users\****\PycharmProjects\Phyton-Project\venv\lib\site-packages\pywinauto\application.py", line 1220, in __getitem__
return self.window(best_match=key)
File "C:\Users\*****\PycharmProjects\Phyton-Project\venv\lib\site-packages\pywinauto\application.py", line 1207, in window
raise AppNotConnected("Please use start or connect before trying "
pywinauto.application.AppNotConnected: Please use start or connect before trying anything else
Process finished with exit code
ok I decided to create a new function to click on the window - so far I can find the window, set the focus but I can't click enter or any key
import win32con
import win32gui
import pyautogui as pg
from pywinauto.win32structures import HANDLE
toplist = []
winlist = []
def enum_callback(hwnd, results): winlist.append((hwnd, win32gui.GetWindowText(hwnd)))
win32gui.EnumWindows(enum_callback, toplist)
services = [(hwnd, title) for hwnd, title in winlist if 'system32' in title.lower()]
# just grab the first window that matches
services = services[0]
# use the window handle to set focus
win32gui.SetForegroundWindow(services[0])
win32gui.GetForegroundWindow()
# print windows title
print("Windows Title")
print(services)
print('FOUND')
win32gui.EnableWindow(1, True)
win32api.SendMessage(services, win32con.WM_KEYDOWN, win32con.VK_F11, 0)
print('Window Close')
Response:
C:\Users\jromero\PycharmProjects\Phyton-Project\venv\Scripts\python.exe C:/Users/jromero/PycharmProjects/Phyton-Project/click-enter.py
Windows Title
Traceback (most recent call last):
(334514, 'C:\\WINDOWS\\system32\\cmd.exe')
File "C:/Users/jromero/PycharmProjects/Phyton-Project/click-enter.py", line 32, in <module>
FOUND
win32api.SendMessage(services, win32con.WM_KEYDOWN, win32con.VK_F11, 0)
TypeError: The object is not a PyHANDLE object
Process finished with exit code 1
The program was working fine a few days ago, and it just stopped today. Not a single letter has been changed. One of my troubleshooting steps was to remove the file 'output1.mp3' and check if it will work that way, but it didn't. Another thing is that when it wasn't printing out the error, it would continue to play just this one sound file, whether or not it said the correct thing. Here's the latest error I got:
Traceback (most recent call last):
File "main3.py", line 123, in <module>
start()
File "main3.py", line 117, in start
tts(say)
File "main3.py", line 24, in tts
play('output1.mp3')
File "C:\Program Files (x86)\Python36-32\lib\site-packages\playsound.py", line 35, in _playsoundWin
winCommand('open "' + sound + '" alias', alias)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\playsound.py", line 31, in winCommand
raise PlaysoundException(exceptionMessage)
playsound.PlaysoundException:
Error 275 for command:
open "output1.mp3" alias playsound_0.8842337577803419
Cannot find the specified file. Make sure the path and filename are correct.
Here's the code that I use:
import boto3 # used to 'pythonize' Amazon Polly TTS
import speech # speech recognition
from playsound import playsound as play # for playing sound files
import sys # basically only for exiting
# import locator # for determining the location of the user based on IP address
import locator2 # for confirming auto-detected location
# import locator3 # for definitely confirming auto-detection location
import question # module for answering any question
from datetime import datetime # for displaying the date and time
# from time import sleep # sleep (wai()t for) function
from os import popen as read # for reading command outputs "read('COMMAND').read()"
def tts(text):
polly = boto3.client("polly")
spoken_text = polly.synthesize_speech(Text=str(text),
OutputFormat='mp3',
VoiceId='Brian')
with open('output11.mp3', 'wb') as f:
f.write(spoken_text['AudioStream'].read())
f.close()
play('output11.mp3')
def ask(query):
question.question(query)
response = question.answer
print(response)
tts(response)
ai()
def time():
now = datetime.now()
print("Current date and time: ")
print(now.strftime("%H:%M") + "\n")
tts("It is " + now.strftime("%H:%M"))
ai()
def weather():
response = "Based on your IP address, I've detected that you're located in %s. Is that correct?" % locator2.city
print(response)
tts(response)
speech.speech()
if 'yes' in speech.x:
z = read('weather "%s"' % locator2.city).read()
print(z)
tts(z)
ai()
else:
response = 'Please say the name of the city you would like the weather information for. \n'
print(response)
tts(response)
speech.speech()
city = speech.x
wdr = read('weather "%s"' % city).read()
print(wdr)
tts(wdr)
ai()
def thank():
response = "You're very welcome! \n"
print(response)
tts(response)
ai()
def ext():
response = "Goodbye!"
print(response)
tts(response)
sys.exit()
def error():
response = "Invalid request detected, please try again...\n"
print(response)
tts(response)
ai()
def ai():
print('Started listening - Speak!')
speech.speech()
spoken = speech.x
# TODO new commands should be written above this, and their trigger words below :)
question_words = ['?', 'what', 'where', 'when', 'who', 'how', 'why']
if 'time' in spoken:
time()
elif 'weather' in spoken:
weather()
elif any(word in spoken for word in question_words):
ask(spoken)
elif 'thank' in spoken:
thank()
elif 'exit' or 'quit' or 'deactivate' in spoken:
ext()
else:
error()
def start():
say = "Hello! My name is Dave, and I'm your personal assistant. How may I help you today? \n"
print(say)
tts(say)
ai()
if __name__ == '__main__':
try:
start()
except KeyboardInterrupt:
ext()
The speech synthesizer is Amazon Polly. By the way, I was using PyCharm as an IDE and working on Windows 10. When I switch to my Linux machine the speech recognition part breaks.
UPDATE: I was tweaking the code a bit and managed to fix the pyaudio error, but I got another one in the process, this time it was about permissions. Here's the error log:
Traceback (most recent call last):
File "C:/Users/Despot/Desktop/DAv3/main3.py", line 123, in <module>
start()
File "C:/Users/Despot/Desktop/DAv3/main3.py", line 118, in start
ai()
File "C:/Users/Despot/Desktop/DAv3/main3.py", line 96, in ai
time()
File "C:/Users/Despot/Desktop/DAv3/main3.py", line 39, in time
tts("It is " + now.strftime("%H:%M"))
File "C:/Users/Despot/Desktop/DAv3/main3.py", line 21, in tts
with open('output11.mp3', 'wb') as f:
PermissionError: [Errno 13] Permission denied: 'output11.mp3'
UPDATE 2: I have been tikering about and I've found that the issue is only present on my Windows 10 machine, the program works fine on Linux.
Try using the absolute path (complete path) of the audio file instead of the relative path.
For example: "C:/Users/Adam/Desktop/dolphin.wav" instead of just "dolphin.wav"
This worked for me.
playsound libs has a windows directories in them.
If this fail only on Linux you should install playsound lib on the linux machine and then copy only the main3.py to it.
Answer for UPDATE2:
Copy output11.mp3 to another location and change the path to the new location:
with open('CHANGE-THIS-PATH', 'wb') as f:
Also make sure python run as administrator.
I solved this problem by moving the .py and .wav files to a folder less deep inside the file system.
I just renamed the audio file from sample.wav to sample
and then run it as playsound('sample.wav')
Luckily It ran.
Then I came to know that previously it was stored as sample.wav.wav.
So let's keep your audio file name as output.wav and try running your audio file as:
playsound('output.wav.wav') # or else rename it to output and run as
playsound('output.wav') # likewise for other audio formats too
I am trying to create a secure (e.g., SSL/HTTPS) XML-RPC Client Server. The client-server part works perfectly when the required certificates are present on my system; however, when I try to create the certificates during execution, I receive a FileNotFoundError when opening the ssl-wrapped socket even though the certificates are clearly present (because the preceding function created them.)
Why is the FileNotFoundError given when the files are present? (If I simply close and restart the python script no error is produced when opening the socket and everything works with no issue whatsoever.)
I've searched elsewhere for solutions, but the best/closest answer I've found is, perhaps, "race conditions" between creating the certificates and opening them. However, I've tried adding "sleep" to alleviate the possibility of race conditions (as well as running each function individually via a user input menu) with the same error every time.
What I am missing?
Here is a snippet of my code:
import os
import threading
import ssl
from xmlrpc.server import SimpleXMLRPCServer
import certs.gencert as gencert # <---- My python module for generating certs
...
rootDomain = "mydomain"
CERTFILE = "certs/mydomain.cert"
KEYFILE = "certs/mydomain.key"
...
def listenNow(ipAdd, portNum, serverCert, serverKey):
# Create XMLRPC Server, based on ipAdd/port received
server = SimpleXMLRPCServer((ipAdd, portNum))
# **THIS** is what causes the FileNotFoundError ONLY if
# the certificates are created during THE SAME execution
# of the program.
server.socket = ssl.wrap_socket(server.socket,
certfile=serverCert,
keyfile=serverKey,
do_handshake_on_connect=True,
server_side=True)
...
# Start server listening [forever]
server.serve_forever()
...
# Verify Certificates are present; if not present,
# create new certificates
def verifyCerts():
# If cert or key file not present, create new certs
if not os.path.isfile(CERTFILE) or not os.path.isfile(KEYFILE):
# NOTE: This [genert] will create certificates matching
# the file names listed in CERTFILE and KEYFILE at the top
gencert.gencert(rootDomain)
print("Certfile(s) NOT present; new certs created.")
else:
print("Certfiles Verified Present")
# Start a thread to run server connection as a daemon
def startServer(hostIP, serverPort):
# Verify certificates present prior to starting server
verifyCerts()
# Now, start thread
t = threading.Thread(name="ServerDaemon",
target=listenNow,
args=(hostIP,
serverPort,
CERTFILE,
KEYFILE
)
)
t.daemon = True
t.start()
if __name__ == '__main__':
startServer("127.0.0.1", 12345)
time.sleep(60) # <--To allow me to connect w/client before closing
When I run the above, with NO certificates present, this is the error I receive:
$ python3 test.py
Certfile(s) NOT present; new certs created.
Exception in thread ServerDaemon:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "test.py", line 41, in listenNow
server_side=True)
File "/usr/lib/python3.5/ssl.py", line 1069, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python3.5/ssl.py", line 691, in __init__
self._context.load_cert_chain(certfile, keyfile)
FileNotFoundError: [Errno 2] No such file or directory
When I simply re-run the script a second time (i.e., the cert files are already present when it starts, everything runs as expected with NO errors, and I can connect my client just fine.
$ python3 test.py
Certfiles Verified Present
What is preventing the ssl.wrap_socket function from seeing/accessing the files that were just created (and thus producing the FileNotFoundError exception)?
EDIT 1:
Thanks for the comments John Gordon. Here is a copy of gencert.py, courtesy of Atul Varm, found here https://gist.github.com/toolness/3073310
import os
import sys
import hashlib
import subprocess
import datetime
OPENSSL_CONFIG_TEMPLATE = """
prompt = no
distinguished_name = req_distinguished_name
req_extensions = v3_req
[ req_distinguished_name ]
C = US
ST = IL
L = Chicago
O = Toolness
OU = Experimental Software Authority
CN = %(domain)s
emailAddress = varmaa#toolness.com
[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = %(domain)s
DNS.2 = *.%(domain)s
"""
MYDIR = os.path.abspath(os.path.dirname(__file__))
OPENSSL = '/usr/bin/openssl'
KEY_SIZE = 1024
DAYS = 3650
CA_CERT = 'ca.cert'
CA_KEY = 'ca.key'
# Extra X509 args. Consider using e.g. ('-passin', 'pass:blah') if your
# CA password is 'blah'. For more information, see:
#
# http://www.openssl.org/docs/apps/openssl.html#PASS_PHRASE_ARGUMENTS
X509_EXTRA_ARGS = ()
def openssl(*args):
cmdline = [OPENSSL] + list(args)
subprocess.check_call(cmdline)
def gencert(domain, rootdir=MYDIR, keysize=KEY_SIZE, days=DAYS,
ca_cert=CA_CERT, ca_key=CA_KEY):
def dfile(ext):
return os.path.join('domains', '%s.%s' % (domain, ext))
os.chdir(rootdir)
if not os.path.exists('domains'):
os.mkdir('domains')
if not os.path.exists(dfile('key')):
openssl('genrsa', '-out', dfile('key'), str(keysize))
# EDIT 3: mydomain.key gets output here during execution
config = open(dfile('config'), 'w')
config.write(OPENSSL_CONFIG_TEMPLATE % {'domain': domain})
config.close()
# EDIT 3: mydomain.config gets output here during execution
openssl('req', '-new', '-key', dfile('key'), '-out', dfile('request'),
'-config', dfile('config'))
# EDIT 3: mydomain.request gets output here during execution
openssl('x509', '-req', '-days', str(days), '-in', dfile('request'),
'-CA', ca_cert, '-CAkey', ca_key,
'-set_serial',
'0x%s' % hashlib.md5(domain +
str(datetime.datetime.now())).hexdigest(),
'-out', dfile('cert'),
'-extensions', 'v3_req', '-extfile', dfile('config'),
*X509_EXTRA_ARGS)
# EDIT 3: mydomain.cert gets output here during execution
print "Done. The private key is at %s, the cert is at %s, and the " \
"CA cert is at %s." % (dfile('key'), dfile('cert'), ca_cert)
if __name__ == "__main__":
if len(sys.argv) < 2:
print "usage: %s <domain-name>" % sys.argv[0]
sys.exit(1)
gencert(sys.argv[1])
EDIT 2:
Regarding John's comment, "this might mean that those files are being created, but not in the directory [I] expect":
When I have the directory open in another window, I see the files pop up in the correct location during execution. In addition, when running the test.py script a second time with no changes, the files are identified as present in the correct (the same) location. This leads me to believe that file location is not the problem. Thanks for the suggestion. I'll keep looking.
EDIT 3:
I stepped through the gencert.py program, and each of the files are correctly output at the right time during execution. I indicated when exactly they were output within the above file, labeled with "EDIT 3"
While gencert is paused awaiting my input (raw_input), I can open/view/edit the mentioned files in another program with no problem.
In addition, with the first test.py instance running (paused, waiting for user input, just after mydomain.cert appears), I can run a second instance of test.py in another terminal and it sees/uses the files just fine.
Within the first instance, however, if I continue the program it outputs "FileNotFoundError."
The problem contained in the above stems from the use of os.chdir(rootdir) as suggested by John; however, the specifics are slightly different than the created files being in the wrong location. The problem is the current working directory (cwd) of the running program being changed by gencert(). Here are the specifics:
The program is started with test.py, which calls verifyCerts(). At this point the program is running in the current directory of whichever folder test.py is running inside of. Use os.getcwd() to find the current directory at this point. In this case (as an example), it is running in:
/home/name/testfolder/
Next, os.path.isfile() looks for the files "certs/mydomain.cert" and "certs/mydomain.key"; based on the file path above [e.g., the cwd], it is looking for the following files:
/home/name/testfolder/certs/mydomain.cert
/home/name/testfolder/certs/mydomain.key
The above files are not present, so the program executes gencert.gencert(rootDomain) which correctly creates both files as expected in the exact locations mentioned above in number 2.
The problem is indeed the os.chdir() call: When gencert() executes, it uses os.chdir() to change the cwd to "rootdir," which is os.path.abspath(os.path.dirname(__file__)), which is the directory of the current file (gencert.py). Since this file is located a folder deeper, the new cwd becomes:
/home/name/testfolder/certs/
When gencert() finishes execution and the control returns to test.py, the cwd never changes again; the cwd remains as /home/name/testfolder/certs/ even when execution returns to test.py.
Later, when ssl.wrap_socket() tries to find the serverCert and serverKey, it looks for "certs/mydomain.cert" and "certs/mydomain.key" in the cwd, so here is the full path of what it is looking for:
/home/name/testfolder/certs/certs/mydomain.cert
/home/name/testfolder/certs/certs/mydomain.key
These two files are NOT present, so the program correctly returns "FileNotFoundError".
Solution
A) Move the "gencert.py" file to the same directory as "test.py"
B) At the beginning of "gencert.py" add cwd = os.getcwd() to record the original cwd of the program; then, at the very end, add os.chdir(cwd) to change back to the original cwd before ending and giving control back to the calling program.
I went with option 'B', and my program now works flawlessly. I appreciate the assistance from John Gordon to point me toward finding the source of my problem.
I am trying to run a python file from the command line with a single parameter in Ubuntu 12.04. The program works if I simply run it from the IDE and pass the parameter in the code. However, if I call 'python readFromSerial1.py 3' in the command prompt, I get:
Traceback (most recent call last):
File "readFromSerial1.py", line 62, in <module>
main()
File "readFromSerial1.py", line 6, in main
readDataFromUSB(time)
File "readFromSerial1.py", line 9, in readDataFromUSB
import usb.core
ImportError: No module named usb.core
I'm a little confused as the module imports correctly if I run from the IDE. I download the pyUSB module and extracted it (its filename is pyusb-1.0.0a3). I then copied this file into
/usr/local/lib/python2.7/site-packages/. Is that the correct procedure? I have a feeling the issue is due to python simply not being able to find the usb module and I need to put it in the correct location. My code is below, and any help would be greatly appreciated:
readFromSerial1.py
import sys
def main():
time = sys.argv[1]
#time = 1
readDataFromUSB(time)
def readDataFromUSB(time):
import usb.core
#find device
dev = usb.core.find(idVendor=0x099e, idProduct=0x0001) #GPS info
#Was the device found?
if dev is None:
raise ValueError('Device not found')
else:
print "Device found!"
#Do this to avoid 'Errno16: Resource is busy'
if dev.is_kernel_driver_active(0):
try:
dev.detach_kernel_driver(0)
except usb.core.USBError as e:
sys.exit("Could not detach kernel driver: %s" % str(e))
#Sets default config
dev.set_configuration()
#Gets default endpoint
endpoint = dev[0][(0,0)][0]
writeObject = open("InputData.txt", "w")
#iterate for time purposes
for i in range(0, (time*6)): #sys.argv is command line variable for time input
data = dev.read(endpoint.bEndpointAddress, endpoint.wMaxPacketSize, 0, 100000)
sret = ''.join([chr(x) for x in data])
writeObject.write(sret);
print sret
'''
newStr = ''.join(sret[7:14])
compareStr = ",*4F"
if (newStr == compareStr):
print "The GPS is not reading in any values right now. Try going somewhere else with better reception."
else:
print sret[7:14]
'''
writeObject.close()
main()
I have a simple python script that updates that statuses of justin.tv streams in my database. It's a Django based web application. This script worked perfectly before I moved it to my production server, but now it has issues with timing out or freezing. I've solved the time out problem by adding try/except blocks and making the script retry, but I still can't figure out the freezing problem.
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs. Some times however, it just locks up for ever. I just can't picture a scenario where python would freeze infinitely. Here is the code for the script that freezes. I'm linking website.networkmanagers below, as well as oauth and the justin.tv python library that I'm using.
import sys, os, socket
LOG = False
def updateStreamInfo():
# Set necessary paths
honstreams = os.path.realpath(os.path.dirname(__file__) + "../../../")
sys.path.append(honstreams)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
# Import necessary moduels
from website.models import Stream, StreamInfo
from website.networkmanagers import get_manager, \
NetworkManagerReturnedErrorException
# Get all streams
streams = Stream.objects.all()
try:
# Loop through them
for stream in streams:
skipstream = False
print 'Checking %s...' % stream.name,
# Get the appropriate network manager and
manager = get_manager(stream.network.name)
# Try to get stream status up to 3 times
for i in xrange(3):
try:
streamOnline = manager.getStreamOnline(stream.name, LOG)
break
except socket.error as e:
code, message = e
# Retry up to 3 times
print 'Error: %s. Retrying...'
# If this stream should be skipped
if(skipstream):
print 'Can\'t connect! Skipping %s' % stream.name
continue
# Skip if status has not changed
if streamOnline == stream.online:
print 'Skipping %s because the status has not changed' % \
stream.name
continue
# Save status
stream.online = streamOnline
stream.save()
print 'Set %s to %s' % (stream.name, streamOnline)
except NetworkManagerReturnedErrorException as e:
print 'Stopped the status update loop:', e
if(__name__ == "__main__"):
if(len(sys.argv) > 1 and sys.argv[1] == "log"):
LOG = True
if(LOG): print "Logging enabled"
updateStreamInfo()
networkmanagers.py
oauth.py
JtvClient.py
Example of the script freezing
foo#bar:/.../honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
Checking cilantrogamer... Skipping cilantrogamer because the status has not changed
| <- caret sits here blinking infinitely
Interesting update
Every time it freezes and I send a keyboard interrupt, it's on the same line in socket.py:
root#husta:/home/honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
^CChecking cilantrogamer...
Traceback (most recent call last):
File "website/scripts/updateStreamStatus.py", line 64, in <module>
updateStreamInfo()
File "website/scripts/updateStreamStatus.py", line 31, in updateStreamInfo
streamOnline = manager.getStreamOnline(stream.name, LOG)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 47, in getStreamOnline
return self.getChannelLive(channelName, log)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 65, in getChannelLive
response = client.get('/stream/list.json?channel=%s' % channelName)
File "/home/honstreams/honstreams/website/JtvClient.py", line 51, in get
return self._send_request(request, token)
File "/home/honstreams/honstreams/website/JtvClient.py", line 90, in _send_request
return conn.getresponse()
File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
response.begin()
File "/usr/lib/python2.6/httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 397, in readline
data = recv(1)
KeyboardInterrupt
Any thoughts?
Have you tried using another application to open that connection? Given that it's an issue in production, perhaps you don't have some firewall issues.
Down in JtvClient.py it uses httplib to handle the connection. Have you tried changing this to use httplib2 instead?
Other than that stab in the dark, I would add a lot of logging statements to this code in order to track what actually happens and where it gets stuck. Then I would make sure that the point where it gets stuck can timeout on the socket (which usually involves either monkeypatching or forking the codebase) so that stuff fails instead of hanging.
You said:
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs.
Wrong. It doesn't freeze on that line because that line is a function call which calls lots of other functions through several levels of other modules. So you do not yet know where the program freezes. Also, that line is NOT the point where the socket timeout occurs. The socket timeout will only occur on a low level socket operation like select or recv which is being called several times in the chain of activity triggered by getStreamOnline.
You need to trace your code in a debugger or add print statements to track down exactly where the hang occurs. It could possibly be an infinite loop in Python but is more likely to be a low-level call to an OS networking function. Until you find the source of the error, you can't do anything.
P.S. the keyboard interrupt is a reasonable clue that the problem is around line 90 in JtvClient.py, so put in some print statements and find out what happens. There may be a stupid loop in there that keeps calling getresponse, or you may be calling it with bad parameters or maybe the network server really is borked. Narrow it down to fewer possibilities.
It turns out this HTTP connection isn't passed a timeout in jtvClient.py
def _get_conn(self):
return httplib.HTTPConnection("%s:%d" % (self.host, self.port))
Changed the last line to
return httplib.HTTPConnection("%s:%d" % (self.host, self.port), timeout=10)
Which solved it