I'm using Heroku Scheduler to run a script outside of my app. The script is in the same folder as the run.py and procfile.
#manager.command
def run_purge():
candidates = models.Candidate.query.all()
print "SECRET KEY --------->", os.environ["APP_SECRET_KEY"]
people_purged = []
for candidate in candidates:
if candidate.status != 0 and candidate.approved == True and over_30_days(candidate.last_status_change):
payload = reactivate_account_link(candidate.email, 'reactivate_account')
send_email("Your account is innactive", "TalentTracker", [candidate.email], payload)
candidate.status = 0
db.session.commit()
people_purged.append(candidate.email)
else:
pass
return send_email("Purge Completed", "TalentTracker", email_to_admin, "purge completed --> {0}".format(people_purged))
The script generates a payload using Flask's It's Dangerous and the payload is received within the views file within the app itself. This works fine locally. However, when I run live it's giving me an "Internal Server Error". Through Print statements, I figured out it's triggering the BadSignature Exception and I'm exactly sure why. My hunch is that it's to do with the secret key being outside of the app but when I print the secret key it's present!
#app.route('/candidates/reactivate_account/<payload>/')
def reactivate_account(payload):
s = get_serializer()
try:
candidate_email = s.loads(payload)[0]
except BadSignature:
print "BAD SIGNATURE", payload, s.loads(payload)
raise
candidate = Candidate.query.filter_by(email=candidate_email).first()
candidate.status += 1
candidate.last_status_change = datetime.datetime.now()
db.session.commit()
commit_to_analytics(candidate.candidate_id, None, 4)
return render_template("test.html")
This is what get_serializer looks like outside of the app.
def get_serializer(secret_key=None):
if secret_key is None:
secret_key = app.secret_key
return URLSafeSerializer(secret_key)
# for getting serialized urls
def reactivate_account_link(candidate_email, path):
s = get_serializer(os.environ["APP_SECRET_KEY"])
loads = [candidate_email]
payload = s.dumps(loads)
return url_for(path, payload=payload, _external=True)
I'm creating it separately outside of the app but the function is the same within. I've tried a version of this where I explicitly call the Secret Key but that didn't work either. Instead of creating it separately should I import it?
----- SECOND UPDATE -----
I got this to work by feeding both get_serialize functions - the one in the app and the one outside - a completely new `secret_key'.
However, when I ran the repr and == on the os.environ["APP_SECRET_KEY"] and app.secret_key the values were the same. The payloads matched too.
When I print this secret key in terminal it appears without backslashes e.g. abcdefghi (which I assumed is correct behaviour). In reality the secret key has backslashes e.g. ab/cd/ef/gh/ij. I'm not sure this is related but thought I would include.
Related
This question already has answers here:
Are global variables thread-safe in Flask? How do I share data between requests?
(4 answers)
Closed 1 year ago.
App works perfectly locally but crushes after deployment- seems like it does so when I get to points where operations on variables take place.
I've looked up other similar questions but with no luck.
I assume I am making some fundamental mistake however I am not able to identify it. I was thinking perhaps something with Flask app session settings and the way requests are handled.
It's my first independent project and I realize that there must be more pythonic way of achieving things however I was just focusing on problem solving with this one.
After deployment app either crushes (NoneType object) or goes into the loop looking like variables are not being set at all- none of which happens when app is tested locally.
Fragment of code (up to the point where app crushes- I don't want to spam entire code here):
from flask import Flask, render_template, request, flash, redirect, url_for
import codecs
import datetime
from sq_search import *
from list_creator import list_creator
app = Flask(__name__)
app.secret_key= 'dev'
# Global variables created- to avoid "not defined error" in certain parts of script-
# depending on user choice
stype=None
namesearch=None
final_name=None
results=None
ticker_result=None
name_result=None
company=None
from_date=None
to_date=None
disable_1st=None
email_name=None
#app.route('/')
def home():
# Setting global variables used in search to None- in case user stops search, goes to
# other page and returns back to search- avoids errors
global stype, namesearch, final_name, results, ticker_result, name_result, company
stype=None
namesearch=None
final_name=None
results=None
ticker_result=None
name_result=None
company=None
return render_template("home.html")
#app.route('/seng/', methods=['POST','GET'])
def seng():
global stype, namesearch, final_name, results, ticker_result, name_result
search_options_title="Chosen search options:"
# Using 'try except' block to avoid errors- in case user presses go back or refreshes the
# page the script will execute further and eventuall assign variables to None letting
# the user start new search rather than returning error
if stype == None:
try:
if request.method=="POST":
global search_options
stype=request.form["stype"]
if stype == 'name':
search_options=[">Chosing by Name"]
return render_template("seng.html", pic1="pic1.html", search_options=search_options_title+"<br><br>", search_by=search_options[0],
choice2="choice2.html")
if stype == 'ticker':
search_options=[">Chosing by ticker"]
return render_template("seng.html", pic1="pic1.html", search_options=search_options_title+"<br><br>", search_by=search_options[0],
choice2="choice2tick.html")
except:
pass
if namesearch==None and stype=='ticker':
try:
if request.method=="POST":
ticker_search=request.form["tickersearch"].upper()
get_ticker(ticker_search)
if ticker_result:
stype=None
return redirect(url_for('final_stage'))
else:
ticker_search=None
notick_error="<p style='font-size: 1.4vw;color: red'>Ticker incorrect! Inster S&P 500 ticker, search again by name or browse all companies from main menu</p>"
return render_template("seng.html", pic1="pic1.html", search_options=search_options_title+"<br><br>", search_by=search_options[0],
choice2="choice2tick.html", notick_error=notick_error)
except:
stype=None
pass
elif namesearch==None and stype=='name':
if len(search_options) > 1: # Delets previously used search from right side menu if there was one already
del search_options[1]
if request.method=="POST":
try:
namesearch=request.form["namesearch"]
if namesearch != None:
get_names(namesearch)
if results:
list_creator(results) # Creates HTML script with drop down list of all matches
search_options.append(namesearch)
number_of_options=f"<br><p style='font-size: 1.3vw'>Number of matching results: {len(results)}</p>"
results=None
namesearch=None
return render_template("seng.html",pic1="pic1.html", pic2="pic2.html", search_options=search_options_title+"<br><br>",
search_by=search_options[0]+"<br><br>", search_name=">Look for: '"+search_options[1]+"'<br>",
number_of_options=number_of_options, choice3="choice3.html")
else:
noname_error= "<br><p style='font-size: 1.4vw;color: red'>No matches- no such company in S&P 500. Broaden the search or browse all companies in main menu</p>"
results=None
namesearch=None
return render_template("seng.html", pic1="pic1.html", search_options=search_options_title+"<br><br>", search_by=search_options[0],
choice2="choice2.html", noname_error=noname_error)
except:
stype=None
namesearch=None
results=None # Setting all variables to None- in case user went back a page during search-
ticker_result=None # otherwise would return an error
final_name=None
name_result=None
if final_name==None:
try:
if request.method=="POST":
final_name=request.form["name_final"]
name_result=get_all_byname(final_name) # Function retrives full data based on final user choice from drop down list
return redirect(url_for('final_stage'))
except:
pass
else:
namesearch=None # Same reason as previously- avoiding errors
stype=None
final_name=None
results=None
ticker_result=None
name_result=None
return render_template("seng.html", choice1="choice1.html")
The interpreter is confused about choosing definition for variable. Why do you keep global in function? Anything outside function is global. In case of switching contexts, you could use Flask session or Flask g.
Official Documentation
hello i guess have problem with client and member config which config should i use as you can see i am inserting json as data when i call get_data it returns with no problem but when i try to use predicate-sql it gives me error "hazelcast.errors.HazelcastSerializationError: Exception from server: com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable de-serializer for type -120. This exception is likely caused by differences in t
he serialization configuration between members or between clients and members."
#app.route('/insert_data/<database_name>/<collection_name>', methods=['POST'])
def insert_data(database_name, collection_name):
client = hazelcast.HazelcastClient(cluster_members=[
url
])
dbname_map = client.get_map(f"{database_name}-{collection_name}").blocking()
if request.json:
received_json_data = request.json
received_id = received_json_data["_id"]
del received_json_data["_id"]
dbname_map.put(received_id, received_json_data)
client.shutdown()
return jsonify()
else:
client.shutdown()
abort(400)
#app.route('/get_data/<database_name>/<collection_name>', methods=['GET'])
def get_all_data(database_name, collection_name):
client = hazelcast.HazelcastClient(cluster_members=[
url
])
dbname_map = client.get_map(f"{database_name}-{collection_name}").blocking()
entry_set = dbname_map.entry_set()
output = dict()
datas = []
for key, value in entry_set:
value['_id'] = key
output = value
datas.append(output)
client.shutdown()
return jsonify({"Result":datas})
#bp.route('/get_query/<database_name>/<collection_name>/<name>', methods=['GET'])
def get_query_result(database_name, collection_name,name):
client = hazelcast.HazelcastClient(cluster_members=[
url
])
predicate_map = client.get_map(f"{database_name}-{collection_name}").blocking()
predicate = and_(sql(f"name like {name}%"))
entry_set = predicate_map.values(predicate)
#entry_set = predicate_map.entry_set(predicate)
send_all_data = ""
for x in entry_set:
send_all_data += x.to_string()
send_all_data += "\n"
print(send_all_data)
# print("Retrieved %s values whose age is less than 30." % len(result))
# print("Entry is", result[0].to_string())
# value=predicate_map.get(70)
# print(value)
return jsonify()
i try to change hazelcast.xml according to hazelcast-full-example.xml but i can't start hazelcast after
the changes and do i really have to use serialization ? hazelcast version:4.1 python:3.9
This is most likely happening because you are putting entries of the type dictionary to the map, which is serialized by the pickle because you didn't specify a serializer for that and the client does not know how to handle that correctly, so it fallbacks to the default serializer. However, since pickle serialization is Python-specific, servers cannot deserialize it and throw such an exception.
There are possible solutions to that, see the https://hazelcast.readthedocs.io/en/stable/serialization.html chapter for details.
I think the most appropriate solution for your use case would be Portable serialization which does not require a configuration change or code on the server-side. See the https://hazelcast.readthedocs.io/en/stable/serialization.html#portable-serialization
BTW, client objects are quite heavyweight, so you shouldn't be creating them on demand like this. You can construct it once in your application and share and use it in your endpoints or business-logic code freely since it is thread-safe. The same applies to the map proxy you get from the client. It can also be re-used.
The aim of this program is just to return the values that are passed from a .cfg configuration file called 'defaults.cfg'.
I totally understand what should be getting passed here and to be honest the code for all intents and purposes is copied from an exercise, but it fails with a 'Keying error: (value)' (all values give the keying error, it's just whatever is first) and I don't know why. I've been unable to find a solution online and the code is the same in principle as a friend's more complicated program running a proper web application and his works just fine.
Apparently using capitals for the config keys is a thing and I've done that and I'm sure I have all the necessary libraries/binaries installed.
I'm doing this on Bash on Windows on Ubuntu.
Thanks in advance for any consideration.
default.cfg
[config]
DEBUG = True
IP_ADDRESS = 0.0.0.0
PORT = 5000
configuration.py
import ConfigParser
from flask import Flask
app = Flask(__name__)
#app.route('/')
def root():
return "Sup! Hollerin' at ya from the configuration testing app"
#app.route('/WTF/')
def tellMeh():
return app.config['PORT']
#app.route('/config/')
def config():
str = []
str.append(app.config['DEBUG'])
str.append('port:'+app.config['PORT'])
str.append('ip_address:'+app.config['IP'])
return '\t'.join(str)
def init(app):
config = ConfigParser.ConfigParser()
try:
config_location = "etc/defaults.cfg"
config.read(config_location)
app.config['DEBUG'] = config.get("config", "DEBUG")
app.config['IP'] = config.get("config", "IP_ADDRESS")
app.config['PORT'] = config.get("config", "PORT")
print "Succesfully read configs from: ", config_location
except:
print "Couldn't read configs from: ", config_location
if __name__ == '__main__':
init(app)
app.run(
host=app.config['IP'],
port=int(app.config['PORT']))
You'll get different behavior from that code depending on how you invoke it.
FLASK_APP=configuration.py flask run will skip the section at the bottom where init(app) is called
python configuration.py will run that section, calling init(app).
You might wish to move the call to init() to right below app = Flask(...).
I was playing around with oauth2 to get a better understanding of it. For this reason, I've installed offlineimap which should act as a third-party app. I've found a nice way to read encrypted credentials here on stackexchange.
Based on the linked post I've modified/copied the following python script:
import subprocess
import os
import json
def passwd(file_name):
acct = os.path.basename(file_name)
path = "/PATHTOFILE/%s" % file_name
args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
try:
return subprocess.check_output(args).strip()
except subprocess.CalledProcessError:
return ""
def oauthpasswd(acct, key):
acct = os.path.basename(acct)
path = "/PATHTOFILE/%s_oauth2.gpg" % acct
args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
try:
return str(json.loads(subprocess.check_output(args).strip())['installed'][key])
except subprocess.CalledProcessError:
return ""
def prime_gpg_agent():
ret = False
i = 1
while not ret:
ret = (passwd("prime.gpg") == "prime")
if i > 2:
from offlineimap.ui import getglobalui
sys.stderr.write("Error reading in passwords. Terminating.\n")
getglobalui().terminate()
i += 1
return ret
prime_gpg_agent()
In the corresponding offlineimaprc file I call the function with the correct arguments:
oauth2_client_id = oauthpasswd('gmail', 'client_id')
oauth2_client_secret = oauthpasswd('gmail', 'client_secret')
oauth2_request_url = https://accounts.google.com/o/oauth2/token
oauth2_refresh_token = passwd('gmail_rf_token.gpg')
Please note in the local file the PATHTOFILE is set correctly. What I've done was downloaded the JSON file from Google including the oauth2 credentials and encrypted it. I've stored the refresh token in a separate file.
However, if I run offlineimap I get an authentication error:
ERROR: While attempting to sync account 'gmail'
('http error', 401, 'Unauthorized', <httplib.HTTPMessage instance at 0x7f488c214320>) (configuration is: {'client_secret': "oauthpasswd('gmail', 'client_secret')", 'grant_type': 'refresh_token', 'refresh_token': "passwd('gmail_rf_token.gpg')", 'client_id': "oauthpasswd('gmail', 'client_id')"})
I've tried then to check the outputs of the two python functions passwd and oauthpasswd in a python interpreter. I get the desired outputs. Even more, I've copied the output from the functions within the python interpreter to the offlineimaprc config file and I was able to sync to Gmail. This implies that there must be a mistake when offlineimap executes the file but I can't see what's wrong.
If I only encrypt my Gmail password everything is working. This means there is something going wrong from the details downloaded from Google (client_id, client_secret and refresh token). As pointed out above, the values itself are correct. I've really copied the output of
oauthpasswd('gmail', 'client_id')
oauthpasswd('gmail', 'client_secret')
passwd('gmail_rf_token.gpg')
from a python console to the offlineimaprc file and it worked.
The problem which happens is the following. According to this answer offlineimap does not allow for encryption of all keys within the offlinemaprc file. That's why the python function never gets evaluated and the wrong strings are handed over.
I'm developing an Android application with backend developed using Tastypie and Django. I have a get request for which I want to optionally be able to retrieve the entire object (with complete related fields, rather than URIs). Below is part of the python code for the resource I'm talking about:
class RideResource(ModelResource):
user = fields.ForeignKey(UserResource, 'driver')
origin = fields.ForeignKey(NodeResource, 'origin', full=True)
destination = fields.ForeignKey(NodeResource, 'destination', full=True)
path = fields.ForeignKey(PathResource, 'path')
# if the request has full_path=1 then we perform a deep query, returning the entire path object, not just the URI
def dehydrate(self, bundle):
if bundle.request.GET.get('full_path') == "1":
self.path.full = True
else:
ride_path = bundle.obj.path
try:
bundle.data['path'] = _Helpers.serialise_path(ride_path)
except ObjectDoesNotExist:
bundle.data['path'] = []
return bundle
As you can see the RideResource has a foreign key pointing to PathResource. I'm using the dehydrate function to be able to inspect if the GET request has a parameter "full_path" set to 1. In that case I set programmatically the path variable to full=True. Otherwise I simply return the path URI.
The thing is that the code seems to work only the second time the GET is performed. I've tested it hundreds of times and, when I perform my GET with full_path=1, even tho it enters the if and sets self.path.full = True, the first time it only returns the URI of the PathResource object. While, if I relaunch the exact same request a second time it works perfectly...
Any idea what's the problem?
EDIT AFTER SOLUTION FOUND THANKS TO #Tomasz Jakub Rup
I finally managed to get it working using the following code:
def full_dehydrate(self, bundle, for_list=False):
self.path.full = bundle.request.GET.get('full_path') == "1"
return super(RideResource, self).full_dehydrate(bundle, for_list)
def dehydrate(self, bundle):
if not bundle.request.GET.get('full_path') == "1":
try:
bundle.data['path'] = _Helpers.serialise_path(bundle.obj.path)
except ObjectDoesNotExist:
bundle.data['path'] = []
return bundle
dehydrate is called after full_dehydrate. Overwrite full_dehydrate function.
def full_dehydrate(self, bundle, for_list=False):
self.path.full = bundle.request.GET.get('full_path') == "1"
return super(RideResource, self).full_dehydrate(bundle, for_list)