We Keep Coding

Python requests, Kerberos and NTLM

I've been recently working on a project in which I need to access a asp.net web API in order to get some data. The way I've been gaining access to this API so far is by manually setting the cookies manually within the code and then using requests to get the information that I need. My task now is to automate this process. I get the cookies by using the Chrome developer tools, in the network tab. Now obviously the cookies change every once in a while so I've been trying to make something that will automatically change the cookies inside.
I should mention that the network at which this is being done is air-gaped and getting python libraries inside is kind of tedious, so I am trying to avoid that. It is also the reason why getting code examples here is very complicated.
The way the log-in process works in this web app is as follows (data from chrome dev tools):
Upon entering the URL there are a bunch of redirects which seem to do nothing.
A request is made to /login.aspx which returns a "set-cookie: 'sessionId=xyz'" header and redirects to /LandingPage.aspx
A request is made to /LandingPage.aspx with said cookie which returns a "set-cookie" header with a bunch of cookies (ASP.NET etc'). These are the cookies that I need in order to make the python script access the API.
What's written above is the browser way of doing things, when I try to imitate this in python requests, I get the first cookie from /login.aspx but when it redirects to /LandingPage.aspx, I get a 401 Unauthorized with the following headers:
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
After having done some reading I understood that these response headers are related to NTLM and Kerberos protocols (side question: if it responds with both headers does it mean that I need to provide both authentications or that either one will suffice?).
Quick google search yielded that after these mentioned responses should follow a request with the Kerberos/NTLM token (which I have no idea how to acquire) in order to get a 200 response. I find this pretty weird considering the browser doesn't make any of these requests and the web app just gives it the cookies without it seemingly transferring any NTLM or Kerberos data.
I've thought of a few ways to overcome this and hopefully you could help me figure out whether this would work.
Trying to get the requests-kerberos or requests-ntlm libraries for python and using those to overcome this problem. I would like your opinion to whether this would work. I am reluctant to use this method though, because of what was mentioned above.
Somehow using PowerShell to get these tokens and then somehow using these tokens in python requests without the above mentioned libraries. But I have no idea if this would work either.
I would very much appreciate anyone who could maybe further explain the process that's happening here in general, and of course would greatly appreciate any help with solving this.
Thank you very much!

Trying to get the requests-kerberos or requests-ntlm libraries for python and using those to overcome this problem. I would like your opinion to whether this would work. I am reluctant to use this method though, because of what was mentioned above.
Yes, requests-kerberos would work. HTTP Negotiate means Kerberos almost 100% of the time.
For Linux I'd slightly prefer requests-gssapi, which is based on a more maintained 'gssapi' backend, but at the moment it's limited to Unix-ish systems only – while requests-kerberos has the advantage of supporting Windows through the 'winkerberos' backend. But it doesn't really matter; both will do the job fine.
Don't use NTLM if you can avoid it. Your domain admins will appreciate being able to turn off NTLM domain-wide as soon as they can.
Somehow using PowerShell to get these tokens and then somehow using these tokens in python requests without the above mentioned libraries. But I have no idea if this would work either.
Technically it's possible, but doing this via PowerShell (or .NET in general) is going the long way around. You can achieve exactly the same thing using Python's sspi module, which talks directly to the actual Windows SSPI interface that handles Kerberos ticket acquisition (and NTLM, for that matter).
(The gssapi module is the Linux equivalent, and the spnego module is a cross-platform wrapper around both.)
You can see a few examples here – OP has a .NET example, the answer has Python.
But keep in mind that Kerberos tokens contain not only the service ticket but also a one time use authenticator (to prevent replay attacks), so you need to get a fresh token for every HTTP request.
So don't reinvent the wheel and just use requests-kerberos, which will automatically call SSPI to get a token whenever needed.
it says that in order for requests-kerberos to work there has to be a TGT cached already on the PC. This program is supposed to run for weeks without being interfered with and to my understanding these tickets expire after about 10 hours.
That's typical for all Kerberos use, not just requests-kerberos specifically.
If you run the app on Windows, from an interactive session, then Windows will automatically renew Kerberos tickets as needed (it keeps your password cached in LSA memory for that purpose). However, don't run long-term tasks in interactive sessions...
If you run the app on Windows, as a service, then it will use the "machine credentials" aka "computer account" (see details), and again LSA will keep the tickets up-to-date.
If you run the app on Linux, then you can create a keytab that stores the client credentials for the application. (This doesn't need domain admin rights, you only need to know the app account's password.)
On Linux there are at least 4 different ways to use a keytab for long-term jobs: k5start (third-party, but common); KRB5_CLIENT_KTNAME (built-in to MIT Kerberos, but only in recent versions); gss-proxy (from RedHat, might already be part of the OS); or a basic cronjob that just re-runs kinit to acquire new tickets every 4-6 hours.
I find this pretty weird considering the browser doesn't make any of these requests and the web app just gives it the cookies without it seemingly transferring any NTLM or Kerberos data.
It likely does, you might be overlooking it.
Note that some SSO systems use JavaScript to dynamically probe for whether the browser has Kerberos authentication properly set up – if the main page really doesn't send a token, then it might be an iframe or an AJAX/XHR request that does.

Accessing MongoDB Database on nodejs server using HTTP GET requests

I have a nodejs server setup on AWS with mongoDB. I want to access the database contents using GET method. There is another application in python which needs to access this database present on AWS. I searched on the internet and came across PycURL but I am not getting how to use it exactly. How to approach with pycURL or what can be an alternate solution?

You can build your restful API that is going to handle those GET requests. You have awesome tutorial (with example that you want on bottom):
https://scotch.io/tutorials/build-a-restful-api-using-node-and-express-4
Edit: If you want phyton code for GET requests there is awesome answer here: Simple URL GET/POST function in Python
Edit 2: Example of how would this work. You first need to code your API how to handle GET request and on what route (example: http://localhost:5000/api/getUsers). Than you want to make GET request to that route using Phyton:
Example:
r = requests.get(url="http://localhost:5000/api/getUsers")

I had a similar problem a while ago, there is a tutorial here here. It can lead you towards your intended direction, the drawback may be that in the tutorial, to issue the http request (if I remember correctly), they used postman but I'm sure you can still use PyCurl.

how to monitor outgoing HTTPS

I'm using an external service to manage my users (called stormpath) who charges me per api call. Lately the api calls has increased greatly and I want to
log to file or to stdout all the outgoing http calls from the server (debian)
In my localhost I've done this with many hacks involving monkey patching the pips using mitmproxy
obviously , I can't do it in my production server.
How can I log the https calls and then grep only the relevant ones ?
Is there a simple plugin for flask that will do set a callback every time there is an outgoing HTTPS request from the Flask service outside ?
in my localhost the mitmproxy output looks like this
trying to get the same output using python tools in my production server .

After checking through the stormpath-sdk-python library it seems all requests goes through the stormpath.http.HttpExecutor class, specifically the request method of the HttpExecutor class.
And it seems they previously had logging available for requests but chose to remove it due to problems with encoding.
An option would be to fork the stormpath-sdk-python library and add logging or a callback hook that will suit you.

Review Board Python Web API

I'm new with using third-party APIs, and I was trying to get the Review Board Web API to work in python.
However, I was confused about 3 things:
from rbtools.api.client import RBClient
client = RBClient('http://localhost:8080/')
root = client.get_root()
My first question: is http://localhost:8080/ the server that is running the ReviewBoard server? Is there some sort of test server that I can use instead of running my own?
Again, I don't have much experience with APIs so I was wondering if I needed to do some form of authentication before making these calls.
Finally, if I must set up my own Review Board server to try out the API. Would it be possible to get some code for a very simple example as to how to make the simplest POST and GET request you can think of with minimal setup if for example my server was running on http://localhost:8080/?
Reference : http://www.reviewboard.org/docs/rbtools/0.5/api/overview/

To answer your first question: the answer seems to be yes although their docs don't make it entirely clear.
Their docs say:
Here is an example of how to instantiate the client, and retrieve the Root List Resource resource:
Before the code snippet your pasted. That makes me think that the url being passed is whatever you're trying to use, e.g., you could have that set up on a networked machine called monty_python running on port 5050, then you would do:
client = RBClient('http://monty_python:5050/')
As for a test server you should check the documentation they have about their Web API.
Their examples don't seem to show any authentication being performed in the overview. If you check in other sections (e.g., Tutorial: Creating a Pull Request) you'll see them demonstrate how to authenticate and what can be done after authenticating.
As for your last question, I'm not 100% sure what you're asking, but you should probably check the docs I found for their Web API

python microframeworks and the requests library

Working on a side project in python that both initiates http requests and has a small web server. It got me thinking - it seems like every python microframework has its own model for accessing properties of the http request (e.g. the underlying request object that you can use to get at the querystring parameters, headers, etc. and the underlying response object for setting the status code, response headers, etc). They all allow you access to the same data and they've all kind of re-invented the wheel.
Are there any microframeworks that use the Request and Response objects from the popular requests library instead of having their own implementation? It seems like the requests library is becoming the canonical way to do http requests in python, so this would make a framework especially minimal. It would also be cool when making apps that essentially glue other services together because forwarding/wrapping requests would be trivial. You could just change the .url attribute on the incoming request and call .prepare() to forward the request (yes, for simple forwarding doing it at the webserver level makes more sense, but you get the idea).
Or if there aren't any frameworks that do this in particular, are there any with similar benefits i.e. the incoming http request object also works as an http client?
Edit: Wanted to point out that this is how the http Request object works in Go, that's partly what inspired my question. From the net/http library "A Request represents an HTTP request received by a server or to be sent by a client."

Flask is based on Werkzeug. Werkzeug itself is using Request which itself is using BaseRequest. But this is not the Requests library.
Note that there was a plan to create an httpcore library using both Requests and Werkzeug, but that seems to have stopped. That said both projects are kicking.
Some people have modified Flask in their apps using Requests