We Keep Coding

IBM RTC Python Create Workitem

I am currently looking for a python script which can help in creating a workitem using the XML payload. I tried RTCClient but it is of not much help for future and hence I am looking for script via Requests library from Python
I tried cURL commands and I was able to create workitem in RTC but when I try to repeat the same via Python Requests, I don't get any luck in achieving it. Below is the snippet I am using to achieve the same. During my last GET, I get HTML error as "Javascript is either disabled or not available in your Browser". I believe my authentication is not working proper via Python whereas the same works fine with cURL
Can anyone help in correcting below syntax
RTCCookieURL = 'https://clmtest:9443/jazz/authenticated/identity'
RTCGetCookie = requests.get(RTCCookieURL, verify=False)
RTCCookies=RTCGetCookie.cookies
print(RTCCookies)
RTCAuthURL = 'https://clmtest:9443/jazz/authenticated/j_security_check'
RTCHeaders = {
'Accept': 'text/xml',
'Content-Type': 'application/x-oslc-cm-change-request+xml'
}
RTCAuth = requests.get(RTCAuthURL, auth=HTTPBasicAuth('uname','pwd'), verify=False, allow_redirects=True)
print(RTCAuth.cookies)
RTCGetCatalog = requests.get('https://clmtest:9443/jazz/oslc/workitems/catalog', verify=False, cookies=RTCAuth.cookies)
print(RTCGetCatalog.content)

I guess you're trying to replicate an example I've seen somewhere using Curl to login in two steps - a GET to collect cookies then a POST (because -d data is contained in the Curl command) to do form authentication, with explicit saving cookies on the GET and applying these cookies to the subsequent command(s).
You should use a requests session because it does all the cookie handling for you.
Reference material is here, see below heading FORM challenge https://jazz.net/wiki/bin/view/Main/NativeClientAuthentication. Properly handled, when making a request which requires login the response indicates this and tells you where to go to authenticate, which is a much better plan than simply hardcoding URLs as your code does and as does the simple example below.
Notes:
One point of care you should take - authentication has a timeout configured on the app server, and doing an explicit login will only work for subsequent requests as long as that timout hasn't expired, and then you'll get a challenge which if you ignore it you will start getting 403 responses. Basically, it's better in general to not use an explicit login but to simply always try to make the actual request you want to make and check the response headers (even when the request gets a 200 as for example the whoami before logging in, see code) to look for the need to authenticate, and then do the login and finally replay the original request which wouldn't have had any effect as authentication was needed. Take the approach of handling every request like this then authentication expiry is automatically handled by reauthenticating.
This code below uses hardcoded URLs so won't work if e.g. the jts context root changes from /jts. In a more robust implementation there are few hardcoded urls - obviously your code needs to know the application's URL e.g. https://myserver.com:9443/ccm1 and then I think only minimal harcoding is needed - such as j_security_check and rootservices - every other url should be found from the rootservices (the project catalog) or from content/headers in responses.
ccm and qm do authentication themselves, but rm delegates to jts for auth - the indicators for authrequired tell you where to go so you don't need to (shouldn't) hardcode the differences.
As noted in the reference material direct login as you attempt doesn't work on Tomcat, only Liberty. Also refer to the other note about replaying the original request for Tomcat.
This code below replicates what I can remember of the Curl login sequence, works for Form login to Liberty using the Liberty user registry and that's what I've tested it against. YMMV with other auth mechanisms and this definitely won't work for Jazz Authorization Server which does different redirect for the login.
import requests
import urllib3
# Disable the InsecureRequestWarning - pointless having warnings when
# accessing temporary servers which only ever have self-signed certs
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
SERVER = "https://clmtest:9443"
JTSCookieURL = SERVER+'/ccm/authenticated/identity'
JTSAuthURL = SERVER+'/ccm/j_security_check'
JTSWHOAMI = SERVER+"/ccm/whoami"
RootServices = SERVER+'/ccm/rootservices'
USERNAME='fred'
PWD = 'fred'
# get a session - use this for *every* request to the server
s = requests.session()
# when not authenticated, whoami returns 200 with HTML :-( - but it's a good test of authentication success
GetWhoAmI = s.get(JTSWHOAMI, verify=False)
print( f"{GetWhoAmI=}" )
# whoami when authenticated returns the user id - this is always on the jts
# or when not authenticated returns html
if GetWhoAmI.content != SERVER+"/jts/users/"+USERNAME:
print( "NOT AUTHENTICATED YET!!!")
else:
raise Exception( "We are authenticated which should never happen before we login" )
#
# This works for (default) form authentication using Liberty user registry
#
# do a get to load up session with cookies
GetCookie = s.get(JTSCookieURL, verify=False)
print( f"{GetCookie=}" )
# do a post to authenticate
Auth = s.post(JTSAuthURL, data = { 'j_username': USERNAME, 'j_password': PWD}, verify=False,allow_redirects=True)
print(f"{Auth=}")
# now you can get protected resource - whoami
GetWhoAmI = s.get(JTSWHOAMI, verify=False)
print( f"{GetWhoAmI=}" )
if GetWhoAmI.text != SERVER+"/jts/users/"+USERNAME:
# print( "NOT AUTHENTICATED!!!")
raise Exception( "NOT AUTHENTICATED" )
else:
print( "HURRAY WE ARE AUTHENTICATED" )
# now you can get protected resources, starting with rootservices
GetRootServices = s.get(RootServices, verify=False)
print( f"{GetRootServices=}" )
#print(GetRootServices.content)

The curl equivalent (for windows, for other OS just remove the rem/echo lines) for /rm is something like (for https://SERVER:9443 and admin user/password myid/mypassword):
rem GET to receive the JazzFormAuth cookie
rem important to use the -L option so the authentication redirects are followed
curl -k -L -c cookies.txt "https://SERVER:9443/jts/authenticated/identity" -v
echo ******************************************************************************************
rem POST to authenticate
rem important to use the -L option so the authentication redirects are followed
curl -k -L -b cookies.txt -c cookies.txt -d "" "https://SERVER:9443/jts/j_security_check?j_username=myid&j_password=mypassword" -v
rem GET a protected resource - if fails, returns 302 (because -L isn't specified on this request, if it fails then redirect to auth isn't followed)
curl -k -b cookies.txt -c cookies.txt "https://SERVER:9443/rm/oslc_rm/catalog" -v -H "OSLC-Core-Version: 2.0" -H "Accept: application/rdf+xml"
and then for further requests always use the -b cookies.txt -c cookies.txt options to provide the session authentication. NOTE the cookies will only be valid until the authentication timeout expires - might be six hours.
This example also works for ccm (Engineering Workflow Manager, EWM, formerly RTC) as well as DOORS Next and should work for Engineering Test Manager (ETM formerly known as RQM)

Making asynchronous HTTP requests from a flask service

I have a couple different needs for asynchrony in my Python 3.6 Flask RESTful web service running under Gunicorn.
1) I'd like for one of my service's routes to be able to send an HTTP request to another HTTP service and, without waiting for the response, send a response back to the client that called my service.
Some example code:
#route
def fire_and_forget():
# Send request to other server without waiting
# for it to send a response.
# Return my own response.
2) I'd like for another one of my service's routes to be able to send 2 or more asynchronous HTTP requests to other HTTP services and wait for them all to reply before my service sends a response.
Some example code:
#route
def combine_results():
# Send request to service A
# Send request to service B
# Wait for both to return.
# Do something with both responses
# Return my own response.
Thanks in advance.
EDIT: I am trying to avoid the additional complexity of using a queue (e.g. celery).

You can use eventlets for the the second use case. It's pretty easy to do:
import eventlet
providers = [EventfulPump(), MeetupPump()]
try:
pool = eventlet.GreenPool()
pile = eventlet.GreenPile(pool)
for each in providers:
pile.spawn(each.get, [], 5, loc) # call the interface method
except (PumpFailure, PumpOverride):
return abort(503)
results = []
for res in pile:
results += res
You can wrap each of your api endpoints in a class that implements a "common interface" (in the above it is the get method) and you can make the calls in parallel. I just place them all in a list.
Your other use case is harder to accomplish in straight python. At least a few years ago you would be forced to introduce some sort of worker process like celery to get something like that done. This question seems to cover all the issues:
Making an asynchronous task in Flask
Perhaps things have changed in flask land?

Flask Blueprint Putting something on session

I was trying to put
session['logged_in'] = True
in the session, but in another blueprint it doesn't persist... Why is that?
Is there any better way to keep something in the session?
Extended:
I have a blueprint giving a form to login. When done and submitted, it will set a session key like above. Then it redirects via
return redirect(url_for('admin.index'))
to admin page where If I call the key via
session.get('logged_in')
I get "None" Instead of the True or False one.

I think I understand your confusion now~
Your flask session won't store anything on the server.
the 'session' dict is filled by the cookies from the client request.
Again. that is:
client make login request to server, and got a [login success] response as well as a [cookies] which contains the !!!sessionINFO!!! you think are stored on the server side.
Next time, you must send the whole cookies to the server again, then your session in the server may have data.
Browser will do this for you.
If you use a local client, say python requests library. Then be sure you are making requests with session (for requests-lib, it's requests.Session())
------------------OLD-------------------------------------
Though not an expert, but the case you described should not have happened.
The session is cookies data encrypted with a secret, if you have gone through the document mentioned by Beqa.
Just set
app.secret = '........'
And use session as a dict.
just FYI,
client request---->server (encrypt your_data 'logged_in' and client_relating_data 'maybe: ip, host or etc.', and put the encrypted info in cookies 'session=....') ------> client (get response with cookies)
client request again -----> server (decrypt the cookie 'session=...' with your secret), find the 'logged_in' data and know you are logged in.)
the cookies is something like below.
So, I'm not sure what's actually your trouble when using session, and put some basic information here. Just hope it helps in case.

How can I get uwsgi to accept POST request without data?

I have a web application that serves post requests, and handles cases when no data or empty data are given. When running the application manually, it behaves properly. But when serving it through uwsgi, a specific case fails, POST requests with no data (empty data works):
$ curl -X POST http://www.example.com/search
curl: (52) Empty reply from server
$ curl -d "" http://www.example.com/search
[{...}]
Is there any uwsgi option I missed that could fix this behavior? How can I get uwsgi to properly forward those requests to my web application?

As #roberto stated, it was a problem in my code. I had this piece of code:
When a user does a post query without data, wheezy will try building self.request.form using environ variables like CONTENT_LENGTH, which here doesn't exist and raises an error. Somehow, the try/except did not work with uwsgi, so I changed:
try:
self.request.form
except:
self.request.form = {}
self.request.files = {}
As follow:
if not self.request.environ.get('CONTENT_LENGTH', False):
self.request.form = {}
self.request.files = {}
And it works fine now

How to check browser cookies support with Pyramid

I would like to know, when is the right moment and how to check the browser cookies support.
I understand I have to check the next request and for instance, with beaker, looking for the session key _creation_time or request.headers['Cookie']... and raise an exception if not found but I don't want to do that or something similar for every request. Some parts of my application don't require cookies, like the home page or info, faq page...
When a user logs out, the session gets deleted or invalidated and I used to redirect to the home view, if I check the session key at that moment, I'll not find it but it doesn't mean there is this issue.
An example I used at the beginning of login view:
try: request.headers['Cookie']
except KeyError:
return HTTPFound(location=request.route_url('home'))
Please also note that if I try to print an error message using request.session.flash(msg, 'error') or use the snippet again at the beginning of the home view and render a message with the template using a control return variable, after logout it will be erroneous displayed.
I am looking for the most elegant way to resolve issue...maybe subscribe to a event?...write down a function to call in some interested view?

There are a few things that could the cause of your problems.
Before I continue... FYI Pyramid uses WebOb to handle request and response objects
WebOb Overview
WebOb Class Documentation
Scenario 1
If you call set_cookie under Pyramid , and then do a redirect, the set_cookie will not be sent. This is because redirects create a new response object.
There are a few ways around this:
The most straightforward is to just copy response headers into the cookie when you raise/return a redirect
return HTTPfound( "/path/to/redirect", headers=[ (k,v) for (k,v)\
in self.request.response.headers.iteritems() if k == 'Set-Cookie'] )
OR
resp = HTTPFound(location='/path/to/redirect')
return self.request.response.merge_cookies(resp)
I should also note that MOST browsers accept cookies on redirects, however Safari does not.
another way is to use pyramid's hooks to convert cookies behind the scenes. i wrote subscribers that automate this. they're on pypi and github. https://github.com/jvanasco/pyramid_subscribers_cookiexfer
Scenario 2
There are two ways of handling sessions in Pyramid. Pyramid has its own session library, and then there is Beaker, which handled sessions for Pylons and has Pyramid support that many people use. I can't speak of pyramid.session, but Beaker has two modes to kill the session:
delete()
Delete the cookie, and clear the session
invalidate()
Clear the contents and start a new session
If you call invalidate(), the Beaker session cookie stays the same and all the session data is cleared -- so you can start storing new data into an empty session object.
If you call delete(), the cookie gets killed as does the session data. If you put new information into the session, IIRC, it will go into a new sessionid / cookie . However, as I noted in the first part above, set_cookie will get called but then thrown out during the redirect. So if you delete() the session and then don't migrate the set_cookie headers... the client will never receive a session identifier.
Some example behaviors of cookies under pyramid
Behavior of redirect
User visits site and is given cookie: SessionId=1
User clicks login
App saves login status to session "1"
App calls set_cookie with "LoggedIn=1"
App calls redirect to /home
Redirect sent, no cookies
User lands on /home
App only sees cookie for "SessionId=1"
Behavior of delete with redirect:
User clicks logout
App calls 'delete()' on session, killing the datastore and placing a set_cookie in request.response to expire the old cookie. if a new sessionid is created, that is sent as well.
If app renders a response, then client receives cookies
If app redirects, client does not receive headers to expire the cookie or set up a new one
Behavior of invalidate with redirect:
User clicks logout
App calls 'invalidate()' on session, killing the datastore
App sets a custom "loggedout=0" cookie
If app renders a response, then client receives cookies
If app redirects:
Client does not receive "loggedout=0" header
Client still has the old session cookie, but it was invalidated/purged on the backend, so they are effectively locked out.
side note: I personally don't like using the request.headers interface -- which handles all headers -- to get at cookies. I've had better luck with request.cookies -- which returns a dictionary of cookies.