We Keep Coding

Flask Server not responding to HTTP POST Request

I built a simple flask server in order to automate python scripts based on an HTTP POST Request coming in from an outside service. This service sends these requests in response to events that we have created based on data conditions. All the flask server needs to do is parse the requests, and take the numerical value stored in the request and use it to run the corresponding python script. At one point, this system was working consistently. Fast forward a few months, the server was no longer working when tried again. No changes were made to the code. According to wireshark, The computer hosting the flask server is receiving the request to the correct port, but the request is now timing out. The host system is failing to respond to the request. Any Idea what is going on here? The firewall has temporarily been turned off.
Alternatively, is there another better package to achieve this goal with?
from flask import Flask, request
import threading
import runpy
app = Flask(__name__)
#app.route('/', methods=['POST'])
def PostHandler():
directory = {}
with open(r"M:\redacted") as f:
for line in f:
(key,val) = line.split()
directory[int(key)] = val
print(directory)
path = r"M:\redacted"
content = request.json
content = content['value']
print(content)
sel_script = int(content)
print(directory[sel_script])
runpy.run_path(path_name=path + directory[sel_script])
return
app.run(host="10.244.xx.xx", port=8080, threaded=True)

IBM RTC Python Create Workitem

I am currently looking for a python script which can help in creating a workitem using the XML payload. I tried RTCClient but it is of not much help for future and hence I am looking for script via Requests library from Python
I tried cURL commands and I was able to create workitem in RTC but when I try to repeat the same via Python Requests, I don't get any luck in achieving it. Below is the snippet I am using to achieve the same. During my last GET, I get HTML error as "Javascript is either disabled or not available in your Browser". I believe my authentication is not working proper via Python whereas the same works fine with cURL
Can anyone help in correcting below syntax
RTCCookieURL = 'https://clmtest:9443/jazz/authenticated/identity'
RTCGetCookie = requests.get(RTCCookieURL, verify=False)
RTCCookies=RTCGetCookie.cookies
print(RTCCookies)
RTCAuthURL = 'https://clmtest:9443/jazz/authenticated/j_security_check'
RTCHeaders = {
'Accept': 'text/xml',
'Content-Type': 'application/x-oslc-cm-change-request+xml'
}
RTCAuth = requests.get(RTCAuthURL, auth=HTTPBasicAuth('uname','pwd'), verify=False, allow_redirects=True)
print(RTCAuth.cookies)
RTCGetCatalog = requests.get('https://clmtest:9443/jazz/oslc/workitems/catalog', verify=False, cookies=RTCAuth.cookies)
print(RTCGetCatalog.content)

I guess you're trying to replicate an example I've seen somewhere using Curl to login in two steps - a GET to collect cookies then a POST (because -d data is contained in the Curl command) to do form authentication, with explicit saving cookies on the GET and applying these cookies to the subsequent command(s).
You should use a requests session because it does all the cookie handling for you.
Reference material is here, see below heading FORM challenge https://jazz.net/wiki/bin/view/Main/NativeClientAuthentication. Properly handled, when making a request which requires login the response indicates this and tells you where to go to authenticate, which is a much better plan than simply hardcoding URLs as your code does and as does the simple example below.
Notes:
One point of care you should take - authentication has a timeout configured on the app server, and doing an explicit login will only work for subsequent requests as long as that timout hasn't expired, and then you'll get a challenge which if you ignore it you will start getting 403 responses. Basically, it's better in general to not use an explicit login but to simply always try to make the actual request you want to make and check the response headers (even when the request gets a 200 as for example the whoami before logging in, see code) to look for the need to authenticate, and then do the login and finally replay the original request which wouldn't have had any effect as authentication was needed. Take the approach of handling every request like this then authentication expiry is automatically handled by reauthenticating.
This code below uses hardcoded URLs so won't work if e.g. the jts context root changes from /jts. In a more robust implementation there are few hardcoded urls - obviously your code needs to know the application's URL e.g. https://myserver.com:9443/ccm1 and then I think only minimal harcoding is needed - such as j_security_check and rootservices - every other url should be found from the rootservices (the project catalog) or from content/headers in responses.
ccm and qm do authentication themselves, but rm delegates to jts for auth - the indicators for authrequired tell you where to go so you don't need to (shouldn't) hardcode the differences.
As noted in the reference material direct login as you attempt doesn't work on Tomcat, only Liberty. Also refer to the other note about replaying the original request for Tomcat.
This code below replicates what I can remember of the Curl login sequence, works for Form login to Liberty using the Liberty user registry and that's what I've tested it against. YMMV with other auth mechanisms and this definitely won't work for Jazz Authorization Server which does different redirect for the login.
import requests
import urllib3
# Disable the InsecureRequestWarning - pointless having warnings when
# accessing temporary servers which only ever have self-signed certs
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
SERVER = "https://clmtest:9443"
JTSCookieURL = SERVER+'/ccm/authenticated/identity'
JTSAuthURL = SERVER+'/ccm/j_security_check'
JTSWHOAMI = SERVER+"/ccm/whoami"
RootServices = SERVER+'/ccm/rootservices'
USERNAME='fred'
PWD = 'fred'
# get a session - use this for *every* request to the server
s = requests.session()
# when not authenticated, whoami returns 200 with HTML :-( - but it's a good test of authentication success
GetWhoAmI = s.get(JTSWHOAMI, verify=False)
print( f"{GetWhoAmI=}" )
# whoami when authenticated returns the user id - this is always on the jts
# or when not authenticated returns html
if GetWhoAmI.content != SERVER+"/jts/users/"+USERNAME:
print( "NOT AUTHENTICATED YET!!!")
else:
raise Exception( "We are authenticated which should never happen before we login" )
#
# This works for (default) form authentication using Liberty user registry
#
# do a get to load up session with cookies
GetCookie = s.get(JTSCookieURL, verify=False)
print( f"{GetCookie=}" )
# do a post to authenticate
Auth = s.post(JTSAuthURL, data = { 'j_username': USERNAME, 'j_password': PWD}, verify=False,allow_redirects=True)
print(f"{Auth=}")
# now you can get protected resource - whoami
GetWhoAmI = s.get(JTSWHOAMI, verify=False)
print( f"{GetWhoAmI=}" )
if GetWhoAmI.text != SERVER+"/jts/users/"+USERNAME:
# print( "NOT AUTHENTICATED!!!")
raise Exception( "NOT AUTHENTICATED" )
else:
print( "HURRAY WE ARE AUTHENTICATED" )
# now you can get protected resources, starting with rootservices
GetRootServices = s.get(RootServices, verify=False)
print( f"{GetRootServices=}" )
#print(GetRootServices.content)

The curl equivalent (for windows, for other OS just remove the rem/echo lines) for /rm is something like (for https://SERVER:9443 and admin user/password myid/mypassword):
rem GET to receive the JazzFormAuth cookie
rem important to use the -L option so the authentication redirects are followed
curl -k -L -c cookies.txt "https://SERVER:9443/jts/authenticated/identity" -v
echo ******************************************************************************************
rem POST to authenticate
rem important to use the -L option so the authentication redirects are followed
curl -k -L -b cookies.txt -c cookies.txt -d "" "https://SERVER:9443/jts/j_security_check?j_username=myid&j_password=mypassword" -v
rem GET a protected resource - if fails, returns 302 (because -L isn't specified on this request, if it fails then redirect to auth isn't followed)
curl -k -b cookies.txt -c cookies.txt "https://SERVER:9443/rm/oslc_rm/catalog" -v -H "OSLC-Core-Version: 2.0" -H "Accept: application/rdf+xml"
and then for further requests always use the -b cookies.txt -c cookies.txt options to provide the session authentication. NOTE the cookies will only be valid until the authentication timeout expires - might be six hours.
This example also works for ccm (Engineering Workflow Manager, EWM, formerly RTC) as well as DOORS Next and should work for Engineering Test Manager (ETM formerly known as RQM)

Flask session doesn't update consistently with parallel requests

I'm noticing that when requests running in parallel modify Flask's session, only some keys are recorded. This happens both with Flask's default cookie session and with Flask-Session using the Redis backend. The project is not new, but this only became noticeable once many requests were happening at the same time for the same session.
import time
from flask import Flask, session
from flask_session import Session
app = Flask(__name__)
app.secret_key = "example"
app.config["SESSION_TYPE"] = "redis"
Session(app)
#app.route("/set/<value>")
def set_value(value):
"""Simulate long running task."""
time.sleep(1)
session[value] = "done"
return "ok\n"
#app.route("/keys")
def keys():
return str(session.keys()) + "\n"
The following shell script demonstrates the issue. Notice that all the requests complete, but only one key is present in the final listing, and it's different between test runs.
#!/bin/bash
# set session
curl -c 'cookie' http://localhost:5007/keys
# run parallel
curl -b 'cookie' http://localhost:5007/set/key1 && echo "done1" &
curl -b 'cookie' http://localhost:5007/set/key2 && echo "done2" &
curl -b 'cookie' http://localhost:5007/set/key3 && echo "done3" &
wait
# get result
curl -b 'cookie' http://localhost:5007/keys
$ sh test.sh
dict_keys(['_permanent'])
ok
ok
ok
done3
done1
done2
dict_keys(['_permanent', 'key2'])
$ sh test.sh
dict_keys(['_permanent'])
ok
done3
ok
ok
done2
done1
dict_keys(['_permanent', 'key1'])
Why aren't all the keys present after the requests finish?

Cookie-based sessions are not thread safe. Any given request only sees the session cookie sent with it, and only returns the cookie with that request's modifications. This isn't specific to Flask, it's how HTTP requests work.
You issue three requests in parallel. They all read the initial cookie that only contains the _permanent key, send their requests, and get a response that sets a cookie with their specific key. Each response cookie would have the _permanent key and the key_keyN key only. Whichever request finishes last writes to the file, overwriting previous data, so you're left with its cookie only.
In practice this isn't an issue. The session isn't really meant to store data that changes rapidly between requests, that's what a database is for. Things that modify the session, such as logging in, don't happen in parallel to the same session (and are idempotent anyway).
If you're really concerned about this, use a server-side session to store the data in a database. Databases are good at synchronizing writes.
You're already using Flask-Session and Redis, but digging into the Flask-Session implementation reveals why you have this issue. Flask-Session doesn't store each session key separately, it writes a single serialized value with all the keys. So it suffers the same issue as cookie-based sessions: only what was present during that request is put back into Redis, overwriting what happened in parallel.
In this case, it will be better to write your own SessionInterface subclass to store each key individually. You would override save_session to set all keys in session and delete any that aren't present.

Django service on gunicorn POST request is recieved as GET?

I have a Django rest service running on virutal environment on gunicorn server with the following .wsgi file:
import os, sys import site
site.addsitedir('/opt/valuation/env/lib/python2.7/site-packages')
sys.stdout = sys.stderr
os.environ['DJANGO_SETTINGS_MODULE'] = 'valuation.valuationcont.valuation.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
When I do curl POST call the service works perfectly:
curl -H "Content-Type: application/json" -X POST -d '{...}' -u username:password http://localhost:8000/valuation/predict/
But when I do the same request on API gateway using axios, Django service responds my custom GET response ("GET not supported, try POST").
axios({
method: 'post',
url:'http://localhost:8000/valuation/predict',
headers:{
"Content-Type":"application/json",
"Authorization":"Basic [BASE64 ENCODING]"
},
data:{
...
}
}).then(response=>{
console.log(response.data)
}).catch(err=>{
console.log(err.toString())
})
The request is transformed from GET to POST.
This only happens with the django/gunicorn service.
Since I am new to django/gunicorn I think there is something wrong with the .wsgi file. But how come the curl call then works?
Any help appreciated, been struggling with this for a week now.
Edit:
Managed to recreate the same problem in my local machine. axios POST requests using its API are translated into GET.
Using the axios.post(...) method I managed to get 403 and 201. All while POSTMAN works fine.
I have a suspicion that since the POST fails axios API has a default fallback to GET which then doesn't fail and service responds normally ("GET not supported" as is should).
New step to debug this would be to ask, how do I recreate POSTMAN POST call as close as possible in javascript since POSTMAN is working and it is obviously axios that is causing the problems.

You're not using the same URL. In the curl snippet you request http://localhost:8000/valuation/predict/ but in the second you request http://localhost:8000/valuation/predict - without the final slash.
Django by default redirects URLs that don't end in a slash to one that does, and a redirect is always a GET.

How do parse a curl PUT request in Flask-RESTful?

How do I save the data uploaded using a curl command like curl localhost:5000/upload/test.bin --upload-file tmp/test.bin in a Flask-RESTful PUT handler method?
The code in Ron Harlev's answer to Flask-RESTful - Upload image works with the POST request from curl -F "file=#tmp/test.bin" localhost:5000/upload/test.bin (slightly modified below):
def post(self, filepath):
parse = reqparse.RequestParser()
parse.add_argument('file', type=werkzeug.datastructures.FileStorage, location='files')
args = parse.parse_args()
upload_file = args['file']
upload_file.save("/usr/tmp/{}".format(filepath))
return ({'filepath': filepath}, 200)
But if I try to use the code to handle the PUT request from curl --upload-file (changing post to put above, of course) I get: "'NoneType' object has no attribute 'save'". This refers to the second-last line in the above code.
How do I get a handle to the file data uploaded using curl --upload-file, so I can save it to a local file?
Update: This works around the problem: curl --request PUT -F "file=#tmp/test.bin" localhost:5000/upload/test.bin, but I still don't have an answer to my question.

curls docs define --upload-file as a PUT http request https://curl.haxx.se/docs/manpage.html#-T.
I'm not sure of the need to handle this through a restful API and I'm pretty sure that's where curl is causing problems, perhaps the assumption that this must be done through flask-restful is holding you back?
maybe try building this as a vanilla flask endpoint instead, this code should work for you.
from flask import Flask, request, jsonify
...
#app.route('/simpleupload/<string:filepath>', methods=['POST','PUT'])
def flask_upload(filepath):
with open("/tmp/{}".format(filepath), 'wb') as file:
file.write(request.stream.read())
return (jsonify({'filepath': filepath}), 200)