This is my code in Python - Flask. Here I am entering data for a Theatre (Theatre table) and then fetching the respective id and then adding the screen to the respective theatre_id in the Screen table.
The problem is Theatre gets added to the database and I am able to fetch the id. The code for Screen doesn't seem to work (for loop does work if I comment out the session.screen add and commit statement)
And even the rollback doesn't happen if screen commit doesn't happen.
session_theatre = Session_theatre()
session_screen = Session_screen()
id = 1
if request.method == "POST":
if form.validate_on_submit():
name = str(form.name.data)
city = str(form.location.data)
address = str(form.address.data)
no_of_screen = int(form.total_no_screen.data)
if (name !="" and name!=" " and city != "" and city != " " and address != ""and address != " " and no_of_screen != None):
t = Theatre(name,city,address,1)
try:
session_theatre.add(t)
session_theatre.commit()
query = session_theatre.query(Theatre).filter_by(name=name,city =city).all()
for i in query :
id = i
for i in range (0,no_of_screen):
flash(id)
screen = Screen(str(i+1),1,20,1,20,id)
session_screen.add(screen)
session_screen.commit()
flash("Successfully added !!")
except :
session_screen.rollback()
session_theatre.rollback()
flash("Oops something went wrong !!")
finally:
session_screen.close()
session_theatre.close()
else :
flash("Please fill the input")
return render_template('admin/add_theatre.html',form = form)
Screen Model
class Screen(db.Model):
__tablename__ = "screens"
id = db.Column(db.Integer,primary_key=True)
screen_num = db.Column(db.String(1))
seat_row_start = db.Column(db.Integer)
seat_row_end = db.Column(db.Integer)
seat_col_start = db.Column(db.Integer)
seat_col_end = db.Column(db.Integer)
theatre_id = db.Column(db.Integer, db.ForeignKey('theatres.id'))
def __init__(self, screen_num,
seat_row_start,seat_row_end,seat_col_start,seat_col_end,theatre_id):
self.screen_num = screen_num
self.seat_row_start = seat_row_start
self.seat_row_end = seat_row_end
self.seat_col_start = seat_col_start
self.seat_col_end = seat_col_end
self.theatre_id = theatre_id
To get a list from the query for session_theatre you have to add all() at the end:
query = session_theatre.query(Theatre).filter_by(name=name,city =city).all()
The query returns a list of Theatre objects and you can access the id attribute of each Theatre object iterating over the list and accessing the attribute by its name:
for obj in query :
for i in range (0,no_of_screen):
flash(obj.id)
screen = Screen(i+1,1,20,1,20,obj.id)
session_screen.add(screen)
session_screen.commit()
Related
I have a function which search for json files in a directory, parse the file and write data in the database. My problem is writing in database, because it take around 30 minutes. Any idea how can I speed up writting in a database? I have few quite big files to parse, but parsing the file is not a problem it take around 3 minutes. Currently I am using sqlite but in the future I will change it to PostgreSQL.
Here is my function:
def create_database():
with transaction.atomic():
directory = os.fsencode('data/web_files/unzip')
for file in os.listdir(directory):
filename = os.fsdecode(file)
with open('data/web_files/unzip/{}'.format(filename.strip()), encoding="utf8") as f:
data = json.load(f)
cve_items = data['CVE_Items']
for i in range(len(cve_items)):
database_object = DataNist()
try:
impact = cve_items[i]['impact']['baseMetricV2']
database_object.severity = impact['severity']
database_object.exp_score = impact['exploitabilityScore']
database_object.impact_score = impact['impactScore']
database_object.cvss_score = impact['cvssV2']['baseScore']
except KeyError:
database_object.severity = ''
database_object.exp_score = ''
database_object.impact_score = ''
database_object.cvss_score = ''
for vendor_data in cve_items[i]['cve']['affects']['vendor']['vendor_data']:
database_object.vendor_name = vendor_data['vendor_name']
for description_data in cve_items[i]['cve']['description']['description_data']:
database_object.description = description_data['value']
for product_data in vendor_data['product']['product_data']:
database_object.product_name = product_data['product_name']
database_object.save()
for version_data in product_data['version']['version_data']:
if version_data['version_value'] != '-':
database_object.versions_set.create(version=version_data['version_value'])
My models.py:
class DataNist(models.Model):
vendor_name = models.CharField(max_length=100)
product_name = models.CharField(max_length=100)
description = models.TextField()
date = models.DateTimeField(default=timezone.now)
severity = models.CharField(max_length=10)
exp_score = models.IntegerField()
impact_score = models.IntegerField()
cvss_score = models.IntegerField()
def __str__(self):
return self.vendor_name + "-" + self.product_name
class Versions(models.Model):
data = models.ForeignKey(DataNist, on_delete=models.CASCADE)
version = models.CharField(max_length=50)
def __str__(self):
return self.version
I will appreciate if you can give me any advice how can I improve my code.
Okay, given the structure of the data, something like this might work for you.
This is standalone code aside from that .objects.bulk_create() call; as commented in the code, the two classes defined would actually be models within your Django app.
(By the way, you probably want to save the CVE ID as an unique field too.)
Your original code had the misassumption that every "leaf entry" in the affected version data would have the same vendor, which may not be true. That's why the model structure here has a separate product-version model that has vendor, product and version fields. (If you wanted to optimize things a little, you might deduplicate the AffectedProductVersions even across DataNists (which, as an aside, is not a perfect name for a model)).
And of course, as you had already done in your original code, the importing should be run within a transaction (transaction.atomic()).
Hope this helps.
import json
import os
import types
class DataNist(types.SimpleNamespace): # this would actually be a model
severity = ""
exp_score = ""
impact_score = ""
cvss_score = ""
def save(self):
pass
class AffectedProductVersion(types.SimpleNamespace): # this too
# (foreign key to DataNist here)
vendor_name = ""
product_name = ""
version_value = ""
def import_item(item):
database_object = DataNist()
try:
impact = item["impact"]["baseMetricV2"]
except KeyError: # no impact object available
pass
else:
database_object.severity = impact.get("severity", "")
database_object.exp_score = impact.get("exploitabilityScore", "")
database_object.impact_score = impact.get("impactScore", "")
if "cvssV2" in impact:
database_object.cvss_score = impact["cvssV2"]["baseScore"]
for description_data in item["cve"]["description"]["description_data"]:
database_object.description = description_data["value"]
break # only grab the first description
database_object.save() # save the base object
affected_versions = []
for vendor_data in item["cve"]["affects"]["vendor"]["vendor_data"]:
for product_data in vendor_data["product"]["product_data"]:
for version_data in product_data["version"]["version_data"]:
affected_versions.append(
AffectedProductVersion(
data_nist=database_object,
vendor_name=vendor_data["vendor_name"],
product_name=product_data["product_name"],
version_name=version_data["version_value"],
)
)
AffectedProductVersion.objects.bulk_create(
affected_versions
) # save all the version information
return database_object # in case the caller needs it
with open("nvdcve-1.0-2019.json") as infp:
data = json.load(infp)
for item in data["CVE_Items"]:
import_item(item)
I am using Zapier to catch a webhook and use that info for an API post. The action code runs perfectly fine with "4111111111111111" in place of Ccnum in doSale. But when I use the input_data variable and place it in doSale it errors.
Zapier Input Variable:
Zapier Error:
Python code:
import pycurl
import urllib
import urlparse
import StringIO
class gwapi():
def __init__(self):
self.login= dict()
self.order = dict()
self.billing = dict()
self.shipping = dict()
self.responses = dict()
def setLogin(self,username,password):
self.login['password'] = password
self.login['username'] = username
def setOrder(self, orderid, orderdescription, tax, shipping, ponumber,ipadress):
self.order['orderid'] = orderid;
self.order['orderdescription'] = orderdescription
self.order['shipping'] = '{0:.2f}'.format(float(shipping))
self.order['ipaddress'] = ipadress
self.order['tax'] = '{0:.2f}'.format(float(tax))
self.order['ponumber'] = ponumber
def setBilling(self,
firstname,
lastname,
company,
address1,
address2,
city,
state,
zip,
country,
phone,
fax,
email,
website):
self.billing['firstname'] = firstname
self.billing['lastname'] = lastname
self.billing['company'] = company
self.billing['address1'] = address1
self.billing['address2'] = address2
self.billing['city'] = city
self.billing['state'] = state
self.billing['zip'] = zip
self.billing['country'] = country
self.billing['phone'] = phone
self.billing['fax'] = fax
self.billing['email'] = email
self.billing['website'] = website
def setShipping(self,firstname,
lastname,
company,
address1,
address2,
city,
state,
zipcode,
country,
email):
self.shipping['firstname'] = firstname
self.shipping['lastname'] = lastname
self.shipping['company'] = company
self.shipping['address1'] = address1
self.shipping['address2'] = address2
self.shipping['city'] = city
self.shipping['state'] = state
self.shipping['zip'] = zipcode
self.shipping['country'] = country
self.shipping['email'] = email
def doSale(self,amount, ccnumber, ccexp, cvv=''):
query = ""
# Login Information
query = query + "username=" + urllib.quote(self.login['username']) + "&"
query += "password=" + urllib.quote(self.login['password']) + "&"
# Sales Information
query += "ccnumber=" + urllib.quote(ccnumber) + "&"
query += "ccexp=" + urllib.quote(ccexp) + "&"
query += "amount=" + urllib.quote('{0:.2f}'.format(float(amount))) + "&"
if (cvv!=''):
query += "cvv=" + urllib.quote(cvv) + "&"
# Order Information
for key,value in self.order.iteritems():
query += key +"=" + urllib.quote(str(value)) + "&"
# Billing Information
for key,value in self.billing.iteritems():
query += key +"=" + urllib.quote(str(value)) + "&"
# Shipping Information
for key,value in self.shipping.iteritems():
query += key +"=" + urllib.quote(str(value)) + "&"
query += "type=sale"
return self.doPost(query)
def doPost(self,query):
responseIO = StringIO.StringIO()
curlObj = pycurl.Curl()
curlObj.setopt(pycurl.POST,1)
curlObj.setopt(pycurl.CONNECTTIMEOUT,30)
curlObj.setopt(pycurl.TIMEOUT,30)
curlObj.setopt(pycurl.HEADER,0)
curlObj.setopt(pycurl.SSL_VERIFYPEER,0)
curlObj.setopt(pycurl.WRITEFUNCTION,responseIO.write);
curlObj.setopt(pycurl.URL,"https://secure.merchantonegateway.com/api/transact.php")
curlObj.setopt(pycurl.POSTFIELDS,query)
curlObj.perform()
data = responseIO.getvalue()
temp = urlparse.parse_qs(data)
for key,value in temp.iteritems():
self.responses[key] = value[0]
return self.responses['response']
# NOTE: your username and password should replace the ones below
Ccnum = input_data['Ccnum'] #this variable I would like to use in
#the gw.doSale below
gw = gwapi()
gw.setLogin("demo", "password");
gw.setBilling("John","Smith","Acme, Inc.","123 Main St","Suite 200", "Beverly Hills",
"CA","90210","US","555-555-5555","555-555-5556","support#example.com",
"www.example.com")
r = gw.doSale("5.00",Ccnum,"1212",'999')
print gw.responses['response']
if (int(gw.responses['response']) == 1) :
print "Approved"
elif (int(gw.responses['response']) == 2) :
print "Declined"
elif (int(gw.responses['response']) == 3) :
print "Error"
Towards the end is where the problems are. How can I pass the variables from Zapier into the python code?
David here, from the Zapier Platform team. A few things.
First, I think your issue is the one described here. Namely, I believe input_data's values are unicode. So you'll want to call str(input_data['Ccnum']) instead.
Alternatively, if you want to use Requests, it's also supported and is a lot less finicky.
All that said, I would be remiss if I didn't mention that everything in Zapier code steps gets logged in plain text internally. For that reason, I'd strongly recommend against putting credit card numbers, your password for this service, and any other sensitive data through a Code step. A private server that you control is a much safer option.
Let me know if you've got any other questions!
I have a legacy Google App engine code, which is having the following entity classes in Python
class AffiliateParent(db.Model):
name = db.StringProperty(required = True)
class Affiliate(db.Model):
email = db.StringProperty(required = True)
point_gain = db.IntegerProperty()
point_used = db.IntegerProperty()
#feature_upgrade = db.ListProperty(str)
modified_time = db.DateTimeProperty(auto_now = True)
I was wondering, is the following the correct way to generate unique id? Are they guarantee unique within Affiliate table?
affiliate_parent_key = AffiliateParent.all(keys_only=True).filter('name =', 'yancheng').get();
affiliate_parent = db.get(affiliate_parent_key);
# check whether affiliate exist, if not create one
affiliate_parent = db.get(affiliate_parent_key);
q = Affiliate.all()
q.ancestor(affiliate_parent)
q.filter('email =', email)
affiliate = q.get()
if not affiliate:
affiliate = Affiliate(
email = email,
point_gain = 0,
point_used = 0,
parent = affiliate_parent
)
affiliate.put()
# 4503602445942784
# is this unique?
unique_id = affiliate.key().id()
yes, if you don't supply id or key when instantiating the model, then datastore will generate a unique ID and assign it to your entity when you .put() it... thus affiliate.key.id() will be unique
you can also generate unique IDs using allocate_ids(count)
https://cloud.google.com/appengine/docs/python/datastore/functions#allocate_ids
The following is my model:
I have two tables match and team:
class Match(DictModel):
date_time = db.DateTimeProperty()
team1 = db.StringProperty()
team2 = db.StringProperty()
venue = db.StringProperty()
result = db.IntegerProperty()
class Team(DictModel):
tslug = db.StringProperty()
name = db.StringProperty()
matches_played = db.IntegerProperty()
matches_won = db.IntegerProperty()
rating = db.FloatProperty()
At runtime, when a post request is made to one of the handler functions, i want to dynamically associate a team rating with the queryset of Match and send the value, this is how i try to do:
matches = Match.all()
matches.filter('date_time <=', end)
matches.filter('date_time >=', start)
match_dict = functs.create_dict(matches)
self.response.out.write(match_dict)
and i have a custom function to get fetch the rating from the current team, it is as follows:
def to_dict(self):
return dict([(p, unicode(getattr(self, p))) for p in self.properties()])
def create_dict(matches):
lst = []
for m in matches:
t1 = m.team1
t2 = m.team2
te1 = Team.all().filter("name =", t1).get()
te2 = Team.all().filter("name =", t2).get()
m.setattr('rating1', te1.rating)
m.setattr('rating2', te2.rating)
lst.append(m)
data_dict = json.dumps([l.to_dict() for l in lst])
return data_dict
Trouble: i get error in setattr in place of setattr i also tried m.rating1 = te1 and m.rating2 = te2 but even that does not seem to work. Everything else is working flawlessly.
Please help thanks!
The syntax is setattr(m, 'rating1', te1.rating1); but this is no different from m.rating1 = te1.rating1.
May we see the trace back?
I have an ObjectListView that displays information retrieved from an SQLite DB with SQLAlchemy.
def setupOLV(self):
self.loanResultsOlv.SetEmptyListMsg("No Loan Records Found")
self.loanResultsOlv.SetColumns([
ColumnDefn("Date Issued", "left", 100, "date_issued",
stringConverter="%d-%m-%y"),
ColumnDefn("Card Number", "left", 100, "card_id"),
ColumnDefn("Student Number", "left", 100, "person_id"),
ColumnDefn("Forename", "left", 150, "person_fname"),
ColumnDefn("Surname", "left", 150, "person_sname"),
ColumnDefn("Reason", "left", 150, "issue_reason"),
ColumnDefn("Date Due", "left", 100, "date_due",
stringConverter="%d-%m-%y"),
ColumnDefn("Date Returned", "left", 100, "date_returned",
stringConverter="%d-%m-%y")
])
I also have three models, Loan:
class Loan(DeclarativeBase):
"""
Loan model
"""
__tablename__ = "loans"
id = Column(Integer, primary_key=True)
card_id = Column(Unicode, ForeignKey("cards.id"))
person_id = Column(Unicode, ForeignKey("people.id"))
date_issued = Column(Date)
date_due = Column(Date)
date_returned = Column(Date)
issue_reason = Column(Unicode(50))
person = relation("Person", backref="loans", cascade_backrefs=False)
card = relation("Card", backref="loans", cascade_backrefs=False)
Person:
class Person(DeclarativeBase):
"""
Person model
"""
__tablename__ = "people"
id = Column(Unicode(50), primary_key=True)
fname = Column(Unicode(50))
sname = Column(Unicode(50))
and Card:
class Card(DeclarativeBase):
"""
Card model
"""
__tablename__ = "cards"
id = Column(Unicode(50), primary_key=True)
active = Column(Boolean)
I am trying to join the tables (loans and people) in order to retrieve and display the information in my ObjectListView. Here is my query method:
def getQueriedRecords(session, filterChoice, keyword):
"""
Searches the database based on the filter chosen and the keyword
given by the user
"""
qry = session.query(Loan)
if filterChoice == "person":
result = qry.join(Person).filter(Loan.person_id=='%s' % keyword).all()
elif filterChoice == "card":
result = qry.join(Person).filter(Loan.card_id=='%s' % keyword).all()
return result
I can retrieve and display every field stored in the loans table but forename and surname (should be drawn from people table and joined on person.id) are blank in my ObjectListView. I have SQL output on so I can see the query and it is not selecting at all from the people table.
How can I modify my query/ObjectListView to retrieve and display this information. ?
UPDATE: I have created an example script that is runnable here.
You're only querying for a Loan (qry = session.query(Loan)). Why do you expect something else to be in the results besides what's in the SELECT statement?
I admit that I am pretty new to SQLAlchemy myself, but I thought I would share what I use to display results from my queries. I have a program that uses a SQLite DB with 4+ tables and I pull data from 2-3 of them in a single query and display this information in an ObjectListView. I owe Mike Driscoll for his in depth tutorials, particularly wxPython and SqlAlchemy: An Intro to MVC and CRUD.
Here is what I would possibly add/change in your code.
In your model section add a "display" class such as:
def OlvDisplay(object):
def __init__(self, date_issued, card_id, person_id, fname, sname,
issue_reason, date_due, date_returned):
self.date_issued = date_issued
self.card_id = card_id
self.person_id = person_id
self.person_fname = fname
self.person_sname = sname
self.issue_reason = issue_reason
self.date_due = date_due
self.date_returned = date_returned
This display class is used in the convertResults definition below and assists with making sure the data is formatted properly for the ObjectListView.
The adjustment to your existing query function:
def getQueriedRecords(session, filterChoice, keyword):
"""
Searches the database based on the filter chosen and the keyword
given by the user
"""
qry = session.query(Loan)
if filterChoice == "person":
result = qry.join(Person).filter(Loan.person_id=='%s' % keyword).all()
elif filterChoice == "card":
result = qry.join(Person).filter(Loan.card_id=='%s' % keyword).all()
convertedResults = convertResults(result)
return convertedResults
What we're doing here is creating a local variable that is essentially running the conversion definition and storing the results for the next line, which returns those results.
And the "Convertor" function:
def convertResults(results):
finalResults = []
for record in results:
result = OlvDisplay(
record.date_issued,
record.card_id,
record.person_id,
record.person.fname,
record.person.sname,
record.issue_reason,
record.date_due,
record.date_returned
)
finalResults.append(result)
return finalResults
The important part here are the 2 lines:
record.person.fname
record.person.sname
Since we are wanting to pull information from another table using the established relationship it is important to refer to that relationship to actually see the data.
And to populate the ObjectListView Widget:
theOutput = getQueriedRecords(session, filterChoice, keyword)
self.setupOLV.SetObjects(theOutput)
Hope this helps you out.
-MikeS