We Keep Coding

Python, Graphql Mutation and Dango API

I am working on a project that takes signups from a Django form and transfers them to a website, the info is mainly the data ( Name, surname, email...) and some extra-information (tags).
One of the scripts give me the following error in the cronjob_logs;
Traceback (most recent call last): File
"/usr/local/lib/python2.7/dist-packages/django_cron/management/commands/runcrons.py",
line 71, in run_cron_with_cache_check manager.run(force) File
"/usr/local/lib/python2.7/dist-packages/django_cron/init.py", line
215, in run self.msg = self.cron_job.do() File
"/home/django/django_project/ogx/cron.py", line 31, in do ep_id =
get_ep_id(ep.email) File
"/home/django/django_project/ogx/graph_helper.py", line 75, in
get_ep_id ''', {"query": email}) File
"/usr/local/lib/python2.7/dist-packages/graphqlclient/client.py", line
11, in execute return self._send(query, variables) File
"/usr/local/lib/python2.7/dist-packages/graphqlclient/client.py", line
34, in _send raise e HTTPError: HTTP Error 500: Internal Server Error
The script was working normally some time ago, as for the graph_helper.py it is as follows;
def get_ep_id(email):
client = GraphQLClient(
'*this part I took off for confidentiality*')
result = client.execute('''
query Myquery($query: String!){
allPeople(q:$query)
{
data
{
id
full_name
}
}
}
''', {"query": email})
data = json.loads(result)
if len(data['data']['allPeople']['data']) > 0:
return data['data']['allPeople']['data'][0]['id']
else:
return None
The cron.py in question is the following;
class FetchEPsIDs(CronJobBase):
RUN_EVERY_MINS = 30
schedule = Schedule(run_every_mins=RUN_EVERY_MINS)
code = 'ogx.FetchEPsIDs' # a unique code
def do(self):
eps_query = mc_ogx_app.objects.filter(ep_id__isnull=True)
for ep in eps_query:
ep_id = get_ep_id(ep.email)
ep.ep_id = ep_id
ep.save()
As for the second script;
It is meant to update data called tags taken from the form and sent to the website through the API, now the script itself executes properly with no issues but it does not do what it is supposed to; Here you have the Cron...
class UpdateEpsTags(CronJobBase):
RUN_EVERY_MINS = 30
schedule = Schedule(run_every_mins=RUN_EVERY_MINS)
code = 'ogx.UpdateEpsTags' # a unique code
def do(self):
access_token = 'taken out for confidentiality'
eps_query = mc_ogx_app.objects.filter(ep_id__isnull=False, tags_uploaded=False)
for ep in eps_query:
if len(str(ep.ep_id)) >= 2:
tags_list = []
if ep.country_pref is not None:
tags_list.append(ep.country_pref.tag_id)
if ep.career_pref is not None:
tags_list.append(ep.career_pref.tag_id)
first_tags_list = return_user_tag_list(ep)
tags_list = tags_list + first_tags_list
if ep.product_ogv:
tags_list.append([7])
if ep.product_oge:
tags_list.append([9])
if ep.product_ogt:
tags_list.append([8])
try:
update_ep_tags(int(ep.ep_id), tags_list, access_token,ep.chosen_ref)
ep.tags_uploaded = True
ep.save()
except:
ep.save()
Now for the graphQl query in the script, it goes as follows;
def update_ep_tags(person_id, tags_list, token,referral):
client = GraphQLClient(
'taken out for confidentiality')
result = client.execute('''
mutation Mymutation($persons: [Int]!, $tags: [Int]!,$id: ID!, $referral: String!){
bulkTagUpdateForPeople(person_ids:$persons, tag_list_ids:$tags)
{
id
full_name
tag_lists
{
id
name
}
}
updatePerson(id:$id , person:
{
referral_type:$referral
})
{
full_name
referral_type
programmes
{
short_name
}
}
}
''', {"persons": [person_id], "tags": tags_list, "id": person_id, "referral": referral})
return result
Now executing the query on GraphQl I get Nullability mismatch on variable $id and argument id (ID / ID!).

Tried checking the datatypes and also running the query on Insomnia to check if it works, it does work just fine, I do not seem to grasp where the error is coming from

Solved:
What I am basically trying to do is reference using an email, the email is no longer supported by the API and therefor the query just gives a runtime error.

raise NotImplementedError NotImplementedError

I use pycharm to write a python3 web app project using tornado web framework,
The listing service has been built already. I need to build the remaining two components: the user service and the public API layer. The implementation of the listing service can serve as a good starting point to learn more about how to structure a web application using the Tornado web framework.
I am required to use tornado's built in framework for HTTP request.
error occurs at listening ( app.listen(options.port)) when I tried to run the program:
Traceback (most recent call last):
File "D:/Bill/python/Tornado/99-python-exercise-master/listing_service.py", line 203, in <module>
app.listen(options.port)
File "C:\Program Files\Python38\lib\site-packages\tornado\web.py", line 2116, in listen
server.listen(port, address)
File "C:\Program Files\Python38\lib\site-packages\tornado\tcpserver.py", line 152, in listen
self.add_sockets(sockets)
File "C:\Program Files\Python38\lib\site-packages\tornado\tcpserver.py", line 165, in add_sockets
self._handlers[sock.fileno()] = add_accept_handler(
File "C:\Program Files\Python38\lib\site-packages\tornado\netutil.py", line 279, in add_accept_handler
io_loop.add_handler(sock, accept_handler, IOLoop.READ)
File "C:\Program Files\Python38\lib\site-packages\tornado\platform\asyncio.py", line 100, in add_handler
self.asyncio_loop.add_reader(fd, self._handle_events, fd, IOLoop.READ)
File "C:\Program Files\Python38\lib\asyncio\events.py", line 501, in add_reader
raise NotImplementedError
NotImplementedError
code:
import tornado.web
import tornado.log
import tornado.options
import sqlite3
import logging
import json
import time
class App(tornado.web.Application):
def __init__(self, handlers, **kwargs):
super().__init__(handlers, **kwargs)
# Initialising db connection
self.db = sqlite3.connect("listings.db")
self.db.row_factory = sqlite3.Row
self.init_db()
def init_db(self):
cursor = self.db.cursor()
# Create table
cursor.execute(
"CREATE TABLE IF NOT EXISTS 'listings' ("
+ "id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,"
+ "user_id INTEGER NOT NULL,"
+ "listing_type TEXT NOT NULL,"
+ "price INTEGER NOT NULL,"
+ "created_at INTEGER NOT NULL,"
+ "updated_at INTEGER NOT NULL"
+ ");"
)
self.db.commit()
class BaseHandler(tornado.web.RequestHandler):
def write_json(self, obj, status_code=200):
self.set_header("Content-Type", "application/json")
self.set_status(status_code)
self.write(json.dumps(obj))
# /listings
class ListingsHandler(BaseHandler):
#tornado.gen.coroutine
def get(self):
# Parsing pagination params
page_num = self.get_argument("page_num", 1)
page_size = self.get_argument("page_size", 10)
try:
page_num = int(page_num)
except:
logging.exception("Error while parsing page_num: {}".format(page_num))
self.write_json({"result": False, "errors": "invalid page_num"}, status_code=400)
return
try:
page_size = int(page_size)
except:
logging.exception("Error while parsing page_size: {}".format(page_size))
self.write_json({"result": False, "errors": "invalid page_size"}, status_code=400)
return
# Parsing user_id param
user_id = self.get_argument("user_id", None)
if user_id is not None:
try:
user_id = int(user_id)
except:
self.write_json({"result": False, "errors": "invalid user_id"}, status_code=400)
return
# Building select statement
select_stmt = "SELECT * FROM listings"
# Adding user_id filter clause if param is specified
if user_id is not None:
select_stmt += " WHERE user_id=?"
# Order by and pagination
limit = page_size
offset = (page_num - 1) * page_size
select_stmt += " ORDER BY created_at DESC LIMIT ? OFFSET ?"
# Fetching listings from db
if user_id is not None:
args = (user_id, limit, offset)
else:
args = (limit, offset)
cursor = self.application.db.cursor()
results = cursor.execute(select_stmt, args)
listings = []
for row in results:
fields = ["id", "user_id", "listing_type", "price", "created_at", "updated_at"]
listing = {
field: row[field] for field in fields
}
listings.append(listing)
self.write_json({"result": True, "listings": listings})
#tornado.gen.coroutine
def post(self):
# Collecting required params
user_id = self.get_argument("user_id")
listing_type = self.get_argument("listing_type")
price = self.get_argument("price")
# Validating inputs
errors = []
user_id_val = self._validate_user_id(user_id, errors)
listing_type_val = self._validate_listing_type(listing_type, errors)
price_val = self._validate_price(price, errors)
time_now = int(time.time() * 1e6) # Converting current time to microseconds
# End if we have any validation errors
if len(errors) > 0:
self.write_json({"result": False, "errors": errors}, status_code=400)
return
# Proceed to store the listing in our db
cursor = self.application.db.cursor()
cursor.execute(
"INSERT INTO 'listings' "
+ "('user_id', 'listing_type', 'price', 'created_at', 'updated_at') "
+ "VALUES (?, ?, ?, ?, ?)",
(user_id_val, listing_type_val, price_val, time_now, time_now)
)
self.application.db.commit()
# Error out if we fail to retrieve the newly created listing
if cursor.lastrowid is None:
self.write_json({"result": False, "errors": ["Error while adding listing to db"]}, status_code=500)
return
listing = dict(
id=cursor.lastrowid,
user_id=user_id_val,
listing_type=listing_type_val,
price=price_val,
created_at=time_now,
updated_at=time_now
)
self.write_json({"result": True, "listing": listing})
def _validate_user_id(self, user_id, errors):
try:
user_id = int(user_id)
return user_id
except Exception as e:
logging.exception("Error while converting user_id to int: {}".format(user_id))
errors.append("invalid user_id")
return None
def _validate_listing_type(self, listing_type, errors):
if listing_type not in {"rent", "sale"}:
errors.append("invalid listing_type. Supported values: 'rent', 'sale'")
return None
else:
return listing_type
def _validate_price(self, price, errors):
# Convert string to int
try:
price = int(price)
except Exception as e:
logging.exception("Error while converting price to int: {}".format(price))
errors.append("invalid price. Must be an integer")
return None
if price < 1:
errors.append("price must be greater than 0")
return None
else:
return price
# /listings/ping
class PingHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
self.write("pong!")
def make_app(options):
return App([
(r"/listings/ping", PingHandler),
(r"/listings", ListingsHandler),
], debug=options.debug)
if __name__ == "__main__":
# Define settings/options for the web app
# Specify the port number to start the web app on (default value is port 6000)
tornado.options.define("port", default=6000)
# Specify whether the app should run in debug mode
# Debug mode restarts the app automatically on file changes
tornado.options.define("debug", default=True)
# Read settings/options from command line
tornado.options.parse_command_line()
# Access the settings defined
options = tornado.options.options
# Create web app
app = make_app(options)
app.listen(options.port)
logging.info("Starting listing service. PORT: {}, DEBUG: {}".format(options.port, options.debug))
# Start event loop
tornado.ioloop.IOLoop.instance().start()
How to fix this problem?

Python 3.8 made a backwards-incompatible change to the asyncio package used by Tornado. Applications that use Tornado on Windows with Python 3.8 must call asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) at the beginning of their main file/function. (as documented on the home page of tornadoweb.org)

"errorMessage": "string argument without an encoding", [duplicate]

This question already has answers here:
TypeError: string argument without an encoding
(3 answers)
Closed 3 years ago.
I'm trying to save password string encrypted in DynamoDb, I get this error.
Response:
{
"errorMessage": "string argument without an encoding",
"errorType": "TypeError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 25, in lambda_handler\n encrypted_password = encrypt(session, plain_text_password, key_alias)\n",
" File \"/var/task/lambda_function.py\", line 11, in encrypt\n Plaintext=bytes(secret)\n"
]
}
This is the code I'm trying to work with.
import boto3
import base64
from botocore.exceptions import ClientError
def encrypt(session, secret, alias):
client = session.client('kms')
ciphertext = client.encrypt(
KeyId=alias,
Plaintext=bytes(secret)
)
return base64.b64encode(ciphertext["CiphertextBlob"])
def lambda_handler(event, context):
plain_text_password = event['password']
username = event['username']
key_alias = 'alias/ProjectKey'
table_name = 'Authentication'
session = boto3.session.Session()
table = boto3.resource('dynamodb').Table(table_name)
encrypted_password = encrypt(session, plain_text_password, key_alias)
print('ENCRYPTED STRING: ' + encrypted_password)
item = {
'username':username,
'password':encrypted_password
}
#check if item with the username already exists; if so, update password; else create new item
entry = table.get_item(TableName=table_name, Key={'username':username})
# if an entry with that username already exists, then update its corresponding password
if 'Item' in entry:
print('Item found. Updating password.')
print("entry['Item']" + str(entry['Item']))
response = table.update_item(
Key={
'username': username
},
UpdateExpression="set password = :p",
ExpressionAttributeValues={
':p': encrypted_password
},
ReturnValues="UPDATED_NEW"
)
else:
#if an entry with that username doesn't already exist, then create it
print('Adding new item to table.')
table.put_item(Item=item)
new_entry = table.get_item(TableName=table_name, Key={'username':username})
if 'Item' in new_entry:
print('A new item was inserted in the table.')
else:
print('Failed to insert new item in table')
return 'Function succeeded!'
I tried to run in python 2.7 and python 3 but no go.
I have added Lambda full access and dynamodb full access roles for Lambda and DB respectively and for KMS I have given the same accessess to administrate and key usage.

could you provide more informations (type, ...) on ciphertext["CiphertextBlob"] ?
maybe you just need to convert to bytes, e.g
base64.b64encode(bytes("yourstring", 'utf-8'))
or another way
base64.b64encode(ciphertext["CiphertextBlob"].encode('utf-8'))

Python with sqlalchemy db retrival

I am quite new to python and to sqlalchemy.I have written the following netowrk program.
class SourcetoPort(Base):
""""""
__tablename__ = 'source_to_port'
id = Column(Integer, primary_key=True)
port_no = Column(Integer)
src_address = Column(String)
#----------------------------------------------------------------------
def __init__(self, src_address,port_no):
""""""
self.src_address = src_address
self.port_no = port_no
#
def act_like_switch (self, packet, packet_in):
"""
Implement switch-like behavior.
"""
# Learn the port for the source MAC
print "RECIEVED FROM PORT ",packet_in.in_port , "SOURCE ",packet.src
Session = sessionmaker(bind=engine)
session = Session()
self.mac_to_port[packet.src]=packet_in.in_port
if(self.matrix.get((packet.src,packet.dst))==None):
# create a Session
#print "creating a db session"
#Session = sessionmaker(bind=engine)
#session = Session()
self.matrix[(packet.src,packet.dst)]=0
# Create an entry with address and port
print "creating a new db entry"
entry = SourcetoPort(src_address="packet.src" , port_no=packet_in.in_port)
# Add the record to the session object
session.add(entry)
session.commit()
self.matrix[(packet.src,packet.dst)]+=1
#if self.mac_to_port.get(packet.dst)!=None:
if session.query(SourcetoPort).filter_by(src_address=packet.dst).all():
#send this packet
self.send_packet(packet_in.buffer_id, packet_in.data,self.mac_to_port[packet.dst], packet_in.in_port)
#create a flow modification message
msg = of.ofp_flow_mod()
#set the fields to match from the incoming packet
msg.match = of.ofp_match.from_packet(packet)
#print "SENDING TO PORT " + str(self.mac_to_port[packet.dst]), packet.dst
# send the rule to the switch so that it does not query the controller again.
msg.actions.append(of.ofp_action_output(port=self.mac_to_port[packet.dst]))
# push the rule
self.connection.send(msg)
else:
#print 'flooding the packet '
# Flood this packet out as we don't know about this node.
self.send_packet(packet_in.buffer_id, packet_in.data,
of.OFPP_FLOOD, packet_in.in_port)
In the above code the line
if session.query(SourcetoPort).filter_by(src_address='packet.dst').all():
does not work as I expect it.What I expect is that it should retrive the entry from sqlalchemy database and if it succeeds (as the output is not NONE) it should execute the following code.
When I try to print that line using print
print "session query", session.query(SourcetoPort).filter_by(src_address='packet.dst').all()
The output that I get is
session query []
Am I doing something wrong.It would be great if someone could point this out.
If I change the above line based on the suggestion as
print session.query(SourcetoPort).filter_by(src_address=packet.dst).count()
I get the following error:
creating a new db entry
RECIEVED FROM PORT 2 SOURCE 96:74:ba:a9:92:b9
creating a new db entry
ERROR:core:Exception while handling Connection!PacketIn...
Traceback (most recent call last):
File "ws_thesis/pox/pox/lib/revent/revent.py", line 234, in raiseEventNoErrors
return self.raiseEvent(event, *args, **kw)
File "ws_thesis/pox/pox/lib/revent/revent.py", line 281, in raiseEvent
rv = event._invoke(handler, *args, **kw)
File "ws_thesis/pox/pox/lib/revent/revent.py", line 159, in _invoke
return handler(self, *args, **kw)
File "ws_thesis/pox/tutorial.py", line 137, in _handle_PacketIn
self.act_like_switch(packet, packet_in)
File "ws_thesis/pox/tutorial.py", line 101, in act_like_switch
print session.query(SourcetoPort).filter_by(src_address=packet.dst).count()
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2400, in count
return self.from_self(col).scalar()
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2045, in scalar
ret = self.one()
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2014, in one
ret = list(self)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2057, in __iter__
return self._execute_and_instances(context)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2072, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in execute

If you want to filter items by the contents of a variable, you need to use that variable directly:
session.query(SourcetoPort).filter_by(src_address=packet.dst)
You were instead trying to filter by the string 'packet.dst'.
You made the same mistake when creating the SourcetoPort entry; you probably wanted to store result of the packet.src expression, not the string 'packet.src':
SourcetoPort(src_address=packet.src, port_no=packet_in.in_port)
Note that calling .all() retrieves all matching entries from the database, but you are only using it in a if test. It would be more efficient (potentially much more so) to use .count() instead of .all() to let the database tell us how many items match:
if session.query(SourcetoPort).filter_by(src_address=packet.dst).count():

How to store RSA encrypted data to postgresql by using pycrypto?

I want to use Public/Private key to secure my UserInfo data. I'm new with PyCrypto and PostgreSQL.
I have some items to clarify:
Are Public Key and Private Key constant values?
If it is constant, how can I store it properly?
Lastly but the most important, how can I store my encrypted data to PostgreSQL? and retrieve it for verification?
Would you guide me on how to dealt with Crypto.PublicKey.RSA as method to secure my data.
Environment: Python 2.5, PyCrypto 2.3, PostgreSQL 8.3 UTF-8 encoding
UserInfo model:
class UserInfo(models.Model):
userid = models.TextField(primary_key = True)
password = models.TextField(null = True)
keyword = models.TextField(null = True)
key = models.TextField(null = True, blank = True)
date = models.DateTimeField(null = True, blank = True)
UPDATES1
tests.py:
# -*- encoding:utf-8 -*-
import os
from os.path import abspath, dirname
import sys
from py23.service.models import UserInfo
from Crypto import Random
# Set up django
project_dir = abspath(dirname(dirname(__file__)))
sys.path.insert(0, project_dir)
os.environ['DJANGO_SETTINGS_MODULE'] = 'py23.settings'
from django.test.testcases import TestCase
class AuthenticationTestCase(TestCase):
def test_001_registerUserInfo(self):
import Crypto.PublicKey.RSA
import Crypto.Util.randpool
#pool = Crypto.Util.randpool.RandomPool()
rng = Random.new().read
# craete RSA object by random key
# 1024bit
#rsa = Crypto.PublicKey.RSA.generate(1024, pool.get_bytes)
rsa = Crypto.PublicKey.RSA.generate(1024, rng)
# retrieve public key
pub_rsa = rsa.publickey()
# create RSA object by tuple
# rsa.n is public key?, rsa.d is private key?
priv_rsa = Crypto.PublicKey.RSA.construct((rsa.n, rsa.e, rsa.d))
# encryption
enc = pub_rsa.encrypt("hello", "")
# decryption
dec = priv_rsa.decrypt(enc)
print "private: n=%d, e=%d, d=%d, p=%d, q=%d, u=%d" % (rsa.n, rsa.e, rsa.d, rsa.p, rsa.q, rsa.u)
print "public: n=%d, e=%d" % (pub_rsa.n, pub_rsa.e)
print "encrypt:", enc
print "decrypt:", dec
# text to be signed
text = "hello"
signature = priv_rsa.sign(text, "")
# check if the text has not changed
print pub_rsa.verify(text, signature)
print pub_rsa.verify(text+"a", signature)
# userid = models.TextField(primary_key = True)
# password = models.TextField(null = True)
# keyword = models.TextField(null = True)
# key = models.TextField(null = True, blank = True) is it correct to store the public key here?
# date = models.DateTimeField(null = True, blank = True)
userInfo = UserInfo(userid='test1', password=enc[0], key=pub_rsa.n)
userInfo.save()
print "ok"
result here (failed):
======================================================================
ERROR: test_001_registerUserInfo (py23.service.auth.tests.AuthenticationTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\PIDevelopment\workspace37_pydev\pyh23\py23\service\auth\tests.py", line 64, in test_001_registerUserInfo
userInfo.save()
File "C:\Python25\lib\site-packages\django\db\models\base.py", line 458, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "C:\Python25\lib\site-packages\django\db\models\base.py", line 551, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
File "C:\Python25\Lib\site-packages\django\db\models\manager.py", line 195, in _insert
return insert_query(self.model, values, **kwargs)
File "C:\Python25\lib\site-packages\django\db\models\query.py", line 1524, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "C:\Python25\lib\site-packages\django\db\models\sql\compiler.py", line 788, in execute_sql
cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "C:\Python25\lib\site-packages\django\db\models\sql\compiler.py", line 732, in execute_sql
cursor.execute(sql, params)
File "C:\Python25\lib\site-packages\django\db\backends\util.py", line 15, in execute
return self.cursor.execute(sql, params)
File "C:\Python25\lib\site-packages\django\db\backends\postgresql_psycopg2\base.py", line 44, in execute
return self.cursor.execute(query, args)
DatabaseError: invalid byte sequence for encoding "UTF8": 0x97
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
----------------------------------------------------------------------
Ran 1 test in 90.047s
FAILED (errors=1)

Your problem is that you are trying to store binary data in a text file. Try armoring the data or use bytea (with proper encoding/decoding).