We Keep Coding

SSLError - Certificate verify failed [first question answered, code still not working]

import base64
import math
import bs4
from bs4 import BeautifulSoup
# Define Login Auth
usrPass = "un:pass"
b64Val = base64.b64encode(usrPass)
# Input Direct Link
r = requests.get('https://URL...../api/incident?id=34219"', headers ={"Authorization": "Basic dHJhZ286eGVsb2M=" % b64Val}).text
print(r)
I'm new to Python. What I'm trying to do is web-scrape a page that is an online/in-browser ticketing system. There is API documentation provided that details the 'Get Incident' function and spits out a full request URL and an Authorization Header that converts my login to a base64 code - the problem is that I can't figure out how to format this into my code so it actually works. Currently, I'm getting the 'Not all Arguments converted to STR' err, which I suspect is because I'm misusing '%' somewhere.
Thank you in advance for your help and if there is a better way to execute this code please let me know, I have a feeling there's a better way to go about this that I'm not aware of.

You need to precise where is your formatted code. Here, as b64Val seems to be a string, add %s.
r = requests.get('https://URL...../api/incident?id=34219"', headers={"Authorization": "Basic dHJhZ286eGVsb2M=%s" % b64Val}).text
You can also use fstrings if you are in Python 3:
r = requests.get('https://URL...../api/incident?id=34219"', headers={"Authorization": f"Basic dHJhZ286eGVsb2M={b64Val}"}).text

The purpose of the % operator is to replace special tokens such as %s in a string with actual values . For example:
name = "Andy"
print("Hello %s" % name) # prints Hello Andy
You are getting this error because the string you're using does not have any %s tokens.

POST request remake from Java to python gets null

I am totally new to python flask and encountered a problem when writing some code using the requests and flask modules.
I am working on a project using the web API offered by the Panther platform. The project provided an example using Apache Java.
The source code is as below (see more for details).
public class TestProject {
public static void main(String args[]) throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpPost httppost = new HttpPost("http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?");
StringBody organism = new StringBody("Homo sapiens", ContentType.TEXT_PLAIN);
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
StringBody enrichmentType = new StringBody("process", ContentType.TEXT_PLAIN);
StringBody testType = new StringBody("FISHER", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("FDR", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("BONFERRONI", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("NONE", ContentType.TEXT_PLAIN);
StringBody type = new StringBody("enrichment", ContentType.TEXT_PLAIN);
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("organism", organism)
.addPart("geneList", fileData)
.addPart("enrichmentType", enrichmentType)
.addPart("test_type", testType)
.addPart("type", type)
//.addPart("correction", cor)
.build();
httppost.setEntity(reqEntity);
CloseableHttpResponse response = httpclient.execute(httppost);
try {
//System.out.println("----------------------------------------");
//System.out.println(response.getStatusLine());
HttpEntity resEntity = response.getEntity();
if (resEntity != null) {
System.out.println(IOUtils.toString(resEntity.getContent(), StandardCharsets.UTF_8));
}
EntityUtils.consume(resEntity);
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
The part I am mostly interested in is .addPart("organism", organism) and all the other code with similar structures. They will help pass the parameters from a third-party website to the web API offered by Panther.
I remade the JAVA code into python3 using requests. The code is as follows:
uploadTemp = {'file':open('./app/static/data_temp/temp.txt','rb')}
url="http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
params = {"organism":organism,"geneList":pantherName,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
# or params = {"organism":organism,"geneList":uploadTemp,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
Pantherpost= requests.post(url, params = params)
print(Pantherpost.text)
I am expecting an XML object from the web API including some basic biological information. However, the result I got was null (or \n\n\rnull\n when I print Pantherpost.content)
It seems that the parameters I have got from my own web were not correctly sent to the web API.
In addition to this getting null problem, as a beginner, I am also not quite sure about whether the "geneList" part should be receiving a plain-text object or a file. The manual says it is expecting a file, however, it may have been reformatted into plain-text by this command
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
Anyway, I did try both interpretations: pantherName is a list with name correctly formatted in plain-text and uploadTemp is a .txt file generated for the project. There must be some extra bugs in my code since it returned null in both cases.
Can someone please help out? Thank you very much.

I've found the following issues with your python code:
One. If you want to POST a file using requests, you should use keyword files=.
Two. Keys in files object should match respective parameters of the request (you're using file instead).
Three. You put your parameters in the wrong place of the request by writing params=params.
Function annotation from requests source code:
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:Request.
In example Java code StringBody is used to create parameters, which implies that parameters should be placed inside the body of HTTP request, not query string. So you should use data= keyword instead. If you use params=, output will be null.
SO article on difference between data and params keywords in requests.
So I've spent some time reading thier manual and made a test script:
import requests
url = "http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
filepath = "C:\\data\\YOUR_DATA.txt" # change to your file location
# all required parameters according to manual, except geneList which is a file (see below)
params = { # using defaults from manual
"type": "enrichment",
"organism": "Homo sapiens",
"enrichmentType": "process",
"test_type": "FISHER",
"correction": "FDR",
}
# note that the key here is the name of paramter: geneList
files = {'geneList': open(filepath, 'rb')}
# it outputs null, when 'params=params' is used
r = requests.post(url, data=params, files=files)
print(r.status_code)
print(r.text)
Output:
200
Id Name GeneId raw P-value FDR

Updating records with Python 3 through API on Rails 4 application

I'm working on a Rails 4 / mongoid application which needs to expose APIs for other applications and scripts. I need to be able to update documents in one of the models through an API with Python 3 script. I'm a bit fresh with Python hence asking for help here.
I already found out how to query Rails APIs with Python 3 and urllib but struggling with updates. I was trying to go through Python 3.5 docs for urllib2 but struggling to apply this to my script.
What goes to data and how to add authentication token to headers, which in curl would look something like this
-H 'Authorization:Token token="xxxxxxxxxxxxxxxx"'
-X POST -d "name=A60E001&status=changed"
I would greatly appreciate if somebody explained how to, for example, update status based on name (name is not unique yet but will be). My biggest challenge is the Python side. Once I have the data in params on Rails side I think I can handle it. I think.
I included my model and update action from the controller below.
app/models/experiment.rb
class Experiment
include Mongoid::Document
include Mongoid::Timestamps
field :name, type: String
field :status, type:String
end
app/controllers/api/v1/experiments_controller.rb
module Api
module V1
class ExperimentsController < ActionController::Base
before_filter :restrict_access
...
def update
respond_to do |format|
if #expt_proc.update(expt_proc_params)
format.json { render :show, status: :ok, location: #expt_proc }
else
format.json { render json: #expt_proc.errors, status: :unprocessable_entity }
end
end
end
...
private
def restrict_access
authenticate_or_request_with_http_token do |token, options|
ApiKey.where(access_token: token).exists?
end
end
...

I figured out who to send a PATCH request with Python 3 and update the the record's status by name.
Thanks to this post found out about requests module. I used requests.patch to update the record and it works great.
python code
import requests
import json
url = 'http://0.0.0.0:3000/api/v1/update_expt_queue.json'
payload = {'expt_name' : 'myExperiment1', 'status' : 'finished' }
r = requests.patch(url, payload)
There are two problems remaining:
How to add headers to this request which would allow me to go through token based authentication. At the moment it's disabled. request.patch only takes 2 parameters, it doesn't take headers parameter.
How to access the JSON which is sent in response by Rails API.

Server side includes (ssi) on IIS server, read POST request data fails

My Goal is that my SSI (shtml) will parse POST data parameters and return a long encrypted string which will be based on those parameters.
I am using IIS as the server.
My constraints:
I can't change the url - /document.shtml
The shtml need to return only the base 64 string result.
What i tried:
Use of "exec cgi" in the shtml, this way:
document.shtml:
<!--#exec cgi="/pythonApp/test.py?sdfsdf=sdfsdf" -->
test.py (simplified version , without the encryption logic behind):
import cgitb
cgitb.enable()
import base64,urllib,json,os,sys
import cgi
print "Content-Type: text/plain;charset=utf-8"
print
cl, _ = cgi.parse_header(os.environ['CONTENT_LENGTH'])
if (int(cl)>0):
data = sys.stdin.read(int(cl))
input_j = json.loads(data)
print "AB : ",input_j["AB"]
else:
print "Failed to read POST request"
The problem here, is that if i send post request to .../test.py , then it works,
but if i send the request to document.shtml, the content length will still be positive as described by the environment variable, but the stdin will get an error.
My other idea was to read the POST data , somehow from the shtml itself and pass it to the cgi , this way:
<!--#exec cgi="/pythonApp/test.py?sdfsdf=sdfsdf" -->
The problem , is that i looked over the net and haven't found a way to do it from the shtml.
Any suggestions? am i missing something simple?

I haven't found away to do it using SSI directive , i did understand that the stdin is not passed to the cgi from the main script.
So i found other solution which still answers my constrains.
In the IIS Handler Mapping for SHTML , i configured the executable to be php.
and in the shtml it self i wrote a php code (showing here just the sample that prints the json string):
<?php var_dump($HTTP_RAW_POST_DATA); ?>

How to process JSON in Flask?

I have asked a few questions about this before, but still haven't solved my problem.
I am trying to allow Salesforce to remotely send commands to a Raspberry Pi via JSON (REST API). The Raspberry Pi controls the power of some RF Plugs via an RF Transmitter called a TellStick. This is all setup, and I can use Python to send these commands. All I need to do now is make the Pi accept JSON, then work out how to send the commands from Salesforce.
Someone kindly forked my repo on GitHub, and provided me with some code which should make it work. But unfortunately it still isn't working.
Here is the previous question: How to accept a JSON POST?
And here is the forked repo: https://github.com/bfagundez/RemotePiControl/blob/master/power.py
What do I need to do? I have sent test JSON messages n the Postman extension and in cURL but keep getting errors.
I just want to be able to send various variables, and let the script work the rest out.
I can currently post to a .py script I have with some URL variables, so /python.py?power=on&device=1&time=10&pass=whatever and it figures it out. Surely there's a simple way to send this in JSON?
Here is the power.py code:
# add flask here
from flask import Flask
app = Flask(__name__)
app.debug = True
# keep your code
import time
import cgi
from tellcore.telldus import TelldusCore
core = TelldusCore()
devices = core.devices()
# define a "power ON api endpoint"
#app.route("/API/v1.0/power-on/<deviceId>",methods=['POST'])
def powerOnDevice(deviceId):
payload = {}
#get the device by id somehow
device = devices[deviceId]
# get some extra parameters
# let's say how long to stay on
params = request.get_json()
try:
device.turn_on()
payload['success'] = True
return payload
except:
payload['success'] = False
# add an exception description here
return payload
# define a "power OFF api endpoint"
#app.route("/API/v1.0/power-off/<deviceId>",methods=['POST'])
def powerOffDevice(deviceId):
payload = {}
#get the device by id somehow
device = devices[deviceId]
try:
device.turn_off()
payload['success'] = True
return payload
except:
payload['success'] = False
# add an exception description here
return payload
app.run()

Your deviceID variable is a string, not an integer; it contains a '1' digit, but that's not yet an integer.
You can either convert it explicitly:
device = devices[int(deviceId)]
or tell Flask you wanted an integer parameter in the route:
#app.route("/API/v1.0/power-on/<int:deviceId>", methods=['POST'])
def powerOnDevice(deviceId):
where the int: part is a URL route converter.
Your views should return a response object, a string or a tuple instead of a dictionary (as you do now), see About Responses. If you wanted to return JSON, use the flask.json.jsonify() function:
# define a "power ON api endpoint"
#app.route("/API/v1.0/power-on/<int:deviceId>", methods=['POST'])
def powerOnDevice(deviceId):
device = devices[deviceId]
# get some extra parameters
# let's say how long to stay on
params = request.get_json()
try:
device.turn_on()
return jsonify(success=True)
except SomeSpecificException as exc:
return jsonify(success=False, exception=str(exc))
where I also altered the exception handler to handle a specific exception only; try to avoid Pokemon exception handling; do not try to catch them all!

To retrieve the Json Post values you must use request.json
if request.json and 'email' in request.json:
request.json['email']