We Keep Coding

Post beanstalk queue data to Mssql using python script

Guys I'm currently developing an offline ALPR solution.
So far I've used OpenAlpr software running on Ubuntu. By using a python script I found on StackOverlFlow I'm able to read the beanstalk queue data (plate number & meta data) of the ALPR but I need to send this data from the beanstalk queue to a mssql database. Does anyone know how to export beanstalk queue data or JSON data to the database? The code below is for local-host, how do i modify it to automatically post data to the mssql database? The data in the beanstalk queue is in JSON format [key=value].
The read & write csv was my addition to see if it can save the json data as csv on localdisk
import beanstalkc
import json
from pprint import pprint
beanstalk = beanstalkc.Connection(host='localhost', port=11300)
TUBE_NAME='alprd'
text_file = open('output.csv', 'w')
# For diagnostics, print out a list of all the tubes available in Beanstalk.
print beanstalk.tubes()
# For diagnostics, print the number of items on the current alprd queue.
try:
pprint(beanstalk.stats_tube(TUBE_NAME))
except beanstalkc.CommandFailed:
print "Tube doesn't exist"
# Watch the "alprd" tube; this is where the plate data is.
beanstalk.watch(TUBE_NAME)
while True:
# Wait for a second to get a job. If there is a job, process it and delete it from the queue.
# If not, return to sleep.
job = beanstalk.reserve(timeout=5000)
if job is None:
print "No plates yet"
else:
plates_info = json.loads(job.body)
# Do something with this data (e.g., match a list, open a gate, etc.).
# if 'data_type' not in plates_info:
# print "This shouldn't be here... all OpenALPR data should have a data_type"
# if plates_info['data_type'] == 'alpr_results':
# print "Found an individual plate result!"
if plates_info['data_type'] == 'alpr_group':
print "Found a group result!"
print '\tBest plate: {} ({:.2f}% confidence)'.format(
plates_info['candidates'][0]['plate'],
plates_info['candidates'][0]['confidence'])
make_model = plates_info['vehicle']['make_model'][0]['name']
print '\tVehicle information: {} {} {}'.format(
plates_info['vehicle']['year'][0]['name'],
plates_info['vehicle']['color'][0]['name'],
' '.join([word.capitalize() for word in make_model.split('_')]))
elif plates_info['data_type'] == 'heartbeat':
print "Received a heartbeat"
text_file.write('Best plate')
# Delete the job from the queue when it is processed.
job.delete()
text_file.close()

AFAIK there is no way to directly export data from beanstalkd.
What you have makes sense, that is streaming data out of a tube into a flat file or performing a insert into the DB directly
Given the IOPS beanstalkd can be produced, it might still be a reasonable solution (depends on what performance you are expecting)
Try asking https://groups.google.com/forum/#!forum/beanstalk-talk as well

pySerial works in one script but not another

Now getting output, yet it is wrong Post modified to reflect progress.
I have been reading the documentation as well as following links from this site. I was able to find a script that works in reading data from my Arduino serial output.
It is as follows:
import time
import serial
# configure the serial connections (the parameters differs on the device you are connecting to)
ser = serial.Serial(
port='/dev/ttyACM0',
baudrate=115200,
parity=serial.PARITY_ODD,
stopbits=serial.STOPBITS_TWO,
bytesize=serial.SEVENBITS
)
ser.isOpen()
print 'Enter your commands below.\r\nInsert "exit" to leave the application.'
input=1
while 1 :
# get keyboard input
input = raw_input(">> ")
# Python 3 users
# input = input(">> ")
if input == 'exit':
ser.close()
exit()
else:
# send the character to the device
# (note that I happend a \r\n carriage return and line feed to the characters - this is requested by my device)
ser.write(input + '\r\n')
out = ''
# let's wait one second before reading output (let's give device time to answer)
time.sleep(1)
while ser.inWaiting() > 0:
out += ser.read(1)
if out != '':
print ">>" + out
The output window of Python opens and the code is executed. I am able to type 'exit' to exit or 'read' and a ton of lines from the Arduino serial monitor are now populated in the Python output window.
What I would like is to have the output from the Arduino to constantly be populated in the Python output window. (later I will try to plot the data using Matplotlib)
The code I used to attempt to read everything from the Arduinb is as such:
import time
import serial
# configure the serial connections (the parameters differs on the device you are connecting to)
ser = serial.Serial(
port='/dev/ttyACM0',
baudrate=115200,
parity=serial.PARITY_ODD,
stopbits=serial.STOPBITS_TWO,
bytesize=serial.SEVENBITS
)
ser.isOpen()
input=1
while True:
#First things first, lets wait for data prior to reading
time.sleep(1)
if (ser.inWaiting()>0):
myArduinoData = ser.read()
print myArduinoData
However, when I use the above code the Python execution hangs and I get no output from the serial monitor. This was now corrected by the above code and the community help
The new problem is the output only giving a single digit value and not the two to three digit value output to the Arduino serial monitor.
thanks to #jalo I was able to get the values in question by specifying a byte value in my inWaiting and ser.read statements. The change is as follows:
(ser.inWaiting()>4):
ser = ser.read(4)
Thank you.

Module urllib.request not getting data

I am trying to test this demo program from lynda using Python 3. I am using Pycharm as my IDE. I already added and installed the request package, but when I run the program, it runs cleanly and shows a message "Process finished with exit code 0", but does not show any output from print statement. Where am I going wrong ?
import urllib.request # instead of urllib2 like in Python 2.7
import json
def printResults(data):
# Use the json module to load the string data into a dictionary
theJSON = json.loads(data)
# now we can access the contents of the JSON like any other Python object
if "title" in theJSON["metadata"]:
print(theJSON["metadata"]["title"])
# output the number of events, plus the magnitude and each event name
count = theJSON["metadata"]["count"];
print(str(count) + " events recorded")
# for each event, print the place where it occurred
for i in theJSON["features"]:
print(i["properties"]["place"])
# print the events that only have a magnitude greater than 4
for i in theJSON["features"]:
if i["properties"]["mag"] >= 4.0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"])
# print only the events where at least 1 person reported feeling something
print("Events that were felt:")
for i in theJSON["features"]:
feltReports = i["properties"]["felt"]
if feltReports != None:
if feltReports > 0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"], " reported " + str(feltReports) + " times")
# Open the URL and read the data
urlData = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_day.geojson"
webUrl = urllib.request.urlopen(urlData)
print(webUrl.getcode())
if webUrl.getcode() == 200:
data = webUrl.read()
data = data.decode("utf-8") # in Python 3.x we need to explicitly decode the response to a string
# print out our customized results
printResults(data)
else:
print("Received an error from server, cannot retrieve results " + str(webUrl.getcode()))

Not sure if you left this out on purpose, but this script isn't actually executing any code beyond the imports and function definition. Assuming you didn't leave it out on purpose, you would need the following at the end of your file.
if __name__ == '__main__':
data = "" # your data
printResults(data)
The check on __name__ equaling "__main__" is just so your code is only executing when the file is explicitly run. To always run your printResults(data) function when the file is accessed (like, say, if its imported into another module) you could just call it at the bottom of your file like so:
data = "" # your data
printResults(data)

I had to restart the IDE after installing the module. I just realized and tried it now with "Run as Admin". Strangely seems to work now.But not sure if it was a temp error, since even without restart, it was able to detect the module and its methods.

Your comments re: having to restart your IDE makes me think that pycharm might not automatically detect newly installed python packages. This SO answer seems to offer a solution.
SO answer

Slow downloading with Python Script & TOR (Source Code included)

I am trying to Download html pages with my python script & TOR proxy server. It is running well. But extremely slow & Code is not organized so my IP is renewing most of the time rather downloading pages much. How can I speed the downloading with TOR? How can I organize the code efficiency.
Two script is there. Script1 is executed to download html pages from the website & after get block from the website, Script2 has to be executed to renew the IP with help of TOR proxy. So on... IP gets blocked after few seconds.
Should I lower my threading? How ? Please help me to speed up the process. I am getting only 300-500 html pages per hour.
Here is my Full Code of Script1:
# -*- coding: UTF-8 -*-
import os
import sys
import socks
import socket
import subprocess
import time
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, '127.0.0.1', 9050, True)
socket.socket = socks.socksocket
import urllib2
class WebPage:
def __init__(self, path, country, url, lower=0,upper=9999):
self.dir = str(path)+"/"+ str(country)
self.dir =os.path.join(str(path),str(country))
self.url = url
try:
fin = open(self.dir+"/limit.txt",'r')
limit = fin.readline()
limits = str(limit).split(",")
lower = int(limits[0])
upper = int(limits[1])
fin.close()
except:
fout = open(self.dir+"/limit.txt",'wb')
limits = str(lower)+","+str(upper)
fout.write(limits)
fout.close()
self.process_instances(lower,upper)
def process_instances(self,lower,upper):
try:
os.stat(self.dir)
except:
os.mkdir(self.dir)
for count in range(lower,upper+1):
if count == upper:
print "all downloaded, quitting the app!!"
break
targetURL = self.url+"/"+str(count)
print "Downloading :" + targetURL
req = urllib2.Request(targetURL)
try:
response = urllib2.urlopen(req)
the_page = response.read()
if the_page.find("Your IP suspended")>=0:
print "The IP is suspended"
fout = open(self.dir+"/limit.txt",'wb')
limits = str(count)+","+str(upper)
fout.write(limits)
fout.close()
break
if the_page.find("Too many requests")>=0:
print "Too many requests"
print "Renew IP...."
fout = open(self.dir+"/limit.txt",'wb')
limits = str(count)+","+str(upper)
fout.write(limits)
fout.close()
subprocess.Popen("C:\Users\John\Desktop\Data-Mine\yp\lol\lol2.py", shell=True)
time.sleep(2)
subprocess.call('lol1.py')
if the_page.find("404 error")>=0:
print "the page not exist"
continue
self.saveHTML(count, the_page)
except:
print "The URL cannot be fetched"
execfile('lol1.py')
pass
#continue
raise
def saveHTML(self,count, content):
fout = open(self.dir+"/"+str(count)+".html",'wb')
fout.write(content)
fout.close()
if __name__ == '__main__':
if len(sys.argv) !=6:
print "cannot process!!! Five Parameters are required to run the process."
print "Parameter 1 should be the path where to save the data, eg, /Users/john/data/"
print "Parameter 2 should be the name of the country for which data is collected, eg, japan"
print "Parameter 3 should be the URL from which the data to collect, eg, the website link"
print "Parameter 4 should be the lower limit of the company id, eg, 11 "
print "Parameter 5 should be the upper limit of the company id, eg, 1000 "
print "The output will be saved as the HTML file for each company in the target folder's country"
exit()
else:
path = str(sys.argv[1])
country = str(sys.argv[2])
url = str(sys.argv[3])
lowerlimit = int(sys.argv[4])
upperlimit = int(sys.argv[5])
WebPage(path, country, url, lowerlimit,upperlimit)

TOR is very slow, so it is to be expected that you don't get that much pages per hour. There are however some ways to speed it up. Most notably you could turn on GZIP compression for urllib (see this question for example) to improve the speed a little bit.
TOR as a protocol has rather low bandwidth, because the data needs to be relayed a few times and each relay must use its bandwidth for your request. If data is relayed 6 times - a rather probable number - you would need 6 times the bandwidth. GZIP compression can compress HTML to (in some cases) ~10% of the original size so that will probably speed up the process.

ssh result seems to only be 16bits in python with paramiko

im using paramiko in python 2.7 to connect to a cisco router, send a command and then parse the output of the command in a for loop.
the problem seems to be that the returning result is limited to 65535 characters (16bits). I printed the output and pasted it in an editor to count the characters and thats wht it gave me. Im sure im doing this very wrong because im learning python as I go hehe. here is the code:
import sqlite3
import paramiko
import time
import re
def disable_paging(remote_conn):
'''Disable paging on a Cisco router'''
remote_conn.send("terminal length 0\n")
time.sleep(1)
output = remote_conn.recv(10000)
return output
if __name__ == '__main__':
username = 'user'
password = 'password'
db = sqlite3.connect('cmts-priv.sqlite')
cursor = db.cursor()
cursor.execute('''SELECT ID, ip, hostname, uptime, active, cmtstype FROM cmts''')
all_rows = cursor.fetchall()
print "Producing report. This takes a few seconds so be patient and do not refresh\n"
for row in all_rows:
if (row[4] == 1):
print "Docsis 1.x modems for : " + row[2]
print"\n"
remote_conn_pre = paramiko.SSHClient()
remote_conn_pre.set_missing_host_key_policy(
paramiko.AutoAddPolicy())
ip=row[1]
remote_conn_pre.connect(ip, username=username, password=password)
remote_conn = remote_conn_pre.invoke_shell()
disable_paging(remote_conn)
remote_conn.send("\n")
remote_conn.send("show cable modem docsis version | inc 1\.[10] 1\.[10]\n")
time.sleep(5)
output = remote_conn.recv(100000000)
output = output.split("\n")
remote_conn_pre.close()
#print output
for i in output:
if "0013.11" not in i and "0015.96" not in i and "0015.a2" not in i and "0015.a3" not in i and "0015.a4" not in i and "0015.ce" not in i and "0015.cf" not in i and "0015.d0" not in i:
X = '([0-9a-fA-F]{4}\.[0-9a-fA-F]{4}\.[0-9a-fA-F]{4})'
c = re.compile(X).finditer(i)
if c:
for y in c:
print i[y.start(): y.end()]
print "\n=============================\n"
The result this would give to the output variable would be as follow:
00a0.7373.2a14 C4/0/U1 online 11 1.0 1.0 tdma NB
00a0.7372.ed18 C4/0/U1 online 12 1.0 1.0 tdma NB
00a0.7373.2af2 C4/0/U0 online 20 1.1 1.1 tdma NB
.....
with about 3500 results per router
Then i extract the macs after i filter the ones i dont want and output them in a list. the problem is the result im getting back from the router seems to stop at 16bits when i actually would get much more. it stops at about 1/6th compared to if i produce the output directly in the router cli. I tried to play with timeouts and the sleep and the recv buffer. i just cant figure this one out :(
of and the sqlite db stores a few things bbecuase i use it for a few scripts. what im getting with row[1] is the ip of the router to feed it to the connect string.
Im sure a bunch of you will say im complicating my life as hell but like i said i am learning all of this from google searches hehe and then trying to understand it. this will evolve for sure but for now im really stuck in a pickle with this partial and incomplete result im getting from my routers. help :(

You need to put a loop around your recv, something like:
buff = ''
while not buff.endswith(':~# '):
resp = chan.recv(9999)
buff += resp
print(resp)
See this example: https://gist.github.com/rtomaszewski/3397251