We Keep Coding

How to make my bot skip over urls that don't exist

Hey guys I was wondering if there was a way to make my bot skip invalid urls after 1 try to continue with the for loop but continue doesn't seem to work
def check_valid(stripped_results):
global vstripped_results
vstripped_results = []
for tag in stripped_results:
conn = requests.head("https://" + tag)
conn2 = requests.head("http://" + tag)
status_code = conn.status_code
website_is_up = status_code == 200
if website_is_up:
vstripped_results.append(tag)
else:
continue
stripped results is an array of an unknown amount of domains and Subdomains which is why I have the 'https://' part and tbh I'm not even sure whether my if statement is effective or not.
Any help would be greatly appreciated I don't want to get rate limited by discord anymore from sending so many invalid domains through. :(

This is easy. To check the validity of a URL there exist a python library, namely Validators. This library can be used to validate any URL for if it exist or not. Let's take it step by step.
Firstly,
Here is the documentation link for validators:
https://validators.readthedocs.io/en/latest/
How do you validate a link using validators?
It is simple. Let's work on command line for a moment.
This image shows it. This module gives out boolean result on if it is a valid link or not.
Here for the link of this question it gave out True and when it would be false then it would give you the error.
You can validate it using this syntax:
validators.url('Add your URL variable here')
Remember that this gives boolean value so code for it that way.
So you can use it this way...
I wouldn't be implementing it in your code as I want you to try it yourself once. I would help you with this if you are unable to do it.
Thank You! :)

Try this?
def check_valid(stripped_results):
global vstripped_results
vstripped_results = []
for tag in stripped_results:
conn = requests.head("https://" + tag)
conn2 = requests.head("http://" + tag)
status_code = conn.status_code
website_is_up = status_code == 200
if website_is_up:
vstripped_results.append(tag)
else:
#Do the thing here

2 way to call a resource in Flask-Restful?

I'm learning and using Python, Flask and Flask-Restful for traineeship and I got a question :
Right now I've something like this
class CheckTXT(Resource):
def get(self):
import dns.resolver
dmn = request.args.get('dmn')
answers = dns.resolver.query(dmn, 'TXT')
c = []
for rdata in answers:
for txt_string in rdata.strings:
c.append(txt_info(dmn, txt_string))
end = time.time()
tm = end - start
return lookup("TXT", dmn, c, tm)
and
api.add_resource(CheckTXT, '/lookup/txt')
I'd like to call it by 2 way :
lookup/txt?dmn=stackoverflow.com
/lookup/txt/stackoverflow.com
The first one is working but I don't know how to do the second or even if it's possible.
Someone can help me ?
Thanks for your attention and your patience ! You're helping a young padawan ahah

Yes you can use below as endpoint and possible with flask resful
/lookup/txt/stackoverflow.com
for that you need to add resources like
api.add_resource(CheckTXT, '/lookup/txt/<string:name>') and you can access that field in your implementation like below
class CheckTXT(Resource):
def get(self,name):
print name

Exception handling and blank page

I was reading about Datastore exception handling here and possible errors here but I am having problems implementing it. Here's an example code of what I made.
try:
example.put()
except datastore_errors.TransactionFailedError:
self.response.write('Transaction failed, try again.')
When I try to load my page, nothing appears. I have built a little interface for interacting with my databases but that doesn't show up. If I remove the try/except statements, everything works fine.
What am I doing wrong?
Also why nothing at all is displayed? Shouldn't it display at least the HTML template and just throw an error when I try to put something in the database?
EDIT
Added handler code, more info
What I am doing here is I have a page where I choose a database. Depending on my choice, I open up a template I've made for adding easily instances to that database. So far I've made a template only for one database. I'm also using MathJax library and have check button to check the input before posting it.
When I implement the try/except statement, nothings gets rendered at all. Even the first page, which uses a completely different handler. I want to know why and also what's wrong with the try/except statement.
Hope that helps.
class MainPage(webapp2.RequestHandler):
def get(self):
page = env.get_template('main.html')
self.response.out.write(page.render())
class DatabaseHandler(webapp2.RequestHandler):
def get(self):
database = self.request.get('db')
mathPage = env.get_template('mp_database.html')
menuPage = env.get_template('mi_database.html')
if database == 'mathproblem':
self.response.write(mathPage.render())
elif database == 'menuitem':
self.response.write(menuPage.render())
def post(self):
submit = self.request.get('submit')
if submit == 'Submit':
page = env.get_template('mp_database.html')
problem = self.request.get('problem')
problem_db = MathProblem(task=problem)
problem_db.put()
value = {'added': '\(' + problem + '\)' + ' ' 'added'}
self.response.write(page.render(value))
elif submit == 'Check':
page = env.get_template('mp_database.html')
problem = self.request.get('problem')
values = {'displayed': '$$' + problem + '$$',
'problem': problem, 'text': 'Problem'}
self.response.write(page.render(values))
app = webapp2.WSGIApplication([
('/', MainPage),
('/database', DatabaseHandler),
], debug=True)

sunburnt - how to see the generated query URL

I'm using sunburnt, a python library for talking to Solr. I'm getting some unexpected results and it would help me in debugging if I could see what query was being generated by sunburnt. So instead of doing:
result = query.execute()
I want to do something like
url = query.generate_url()
Is anything like this possible? Are there any hacks that can achieve the same effect?

Found the answer by reading the sunburnt docs more closely. It doesn't get me the exact URL, but is near enough:
params_dict = query.params()

What about adding a print statement like below (this code is from sunburnt 0.5, I think, but it should be very similar no matter what version you're using)?
def select(self, params):
qs = urllib.urlencode(params)
url = "%s?%s" % (self.select_url, qs)
print url #This should spit out the solr url
r, c = self.request(url)
if r.status != 200:
raise SolrError(r, c)
return c

GAE - How Do i edit / update the datastore in python

I have this datastore model
class Project(db.Model)
projectname = db.StringProperty()
projecturl = db.StringProperty()
class Task(db.Model)
project = db.ReferenceProperty(Project)
taskname= db.StringProperty()
taskdesc = db.StringProperty()
How do I edit the value of taskname ? say I have task1 and i want to change it to task1-project

oops sorry, Here is the formatted code:
taskkey = self.request.get("taskkey")
taskid = Task.get(taskkey)
query = db.GqlQuery("SELECt * FROM Task WHERE key =:taskid", taskid=taskid)
if query.count() > 0:
task = Task()
task.taskname = "task1-project"
task.put()
by the way, I get it now. I changed the task=Task() into task = query.get() and it worked.
Thanks for helping by the way.

Given an instance t of Task (e.g. from some get operation on the db) you can perform the alteration you want e.g. by t.taskname = t.taskname + '-project' (if what you want is to "append '-project' to whatever was there before). Eventually, you also probably need to .put t back into the store, of course (but if you make multiple changes you don't need to put it back after each and every change -- only when you're done changing it!-).

Probably the easiest way is to use the admin console. Locally it's:
http://localhost:8080/_ah/admin
and if you've uploaded it, it's the dashboard:
http://appengine.google.com/dashboard?&app_id=******
Here's a link: