We Keep Coding

Storing decryptable passwords for automatied usage

TLDR
I am making a REST Session management solution for industrial automation purposes and need to automatically log into devices to perform configurations.
NOTE:
These devices are 99% of the time going to be isolated to private networks/VPNs (i.e., Will not have a public IP)
Dilemma
I am being tasked with creating a service that can store hardware device credentials so automated configurations (& metrics scraping) can be done. The hardware in question only allows REST Session logins via a POST method where the user and (unencrypted) password are sent in the message body. This returns a Session cookie that my service then stores (in memory).
The service in question consists of:
Linux (Ubuntu 20.04) server
FastAPI python backend
SQLITE3 embedded file DB
Storing Credentials?
My background is not in Security so this is all very new to me but it seems that I should prefer storing a hash (e.g., bcrypt) of my password in my DB for future verification however there will not be any future verification as this is all automated.
This brings me to what seems like is the only solution - hashing the password and using that as the salt to encrypt the password, then storing the hashed password in the DB for decryption purposes later. I know this provides almost 0 security given the DB is compromised but I am at a loss for alternate solutions. Given the DB is embedded, maybe there is some added assurance that the server itself would have to be compromised before the DB itself is compromised? I don't know if there is a technical "right" approach to this, maybe not, however if anyone has any advice I am all ears.

You should consider using a hardware security module (HSM). There are cloud alternatives (like AWS Secrets manager, an encrypted secrets repository based on keys stored in an actual HSM, AWS KMS). Or if your app is not hosted in a public cloud, you can consider buying an actual HSM too, but that's expensive. So it all comes down to the risk you want to accept vs the cost.
You can also consider building architecture to properly protect your secrets. If you build a secure secrets store service and apply appropriate protection (which would be too broad to describe for an answer here), you can at least provide auditing of secret usage, you can implement access control, you can easily revoke secrets, you can monitor usage patterns in that component and so on. Basically your secrets service would act like a very well protected "HSM", albeit it might not involve specialized hardware at all. This would not guarantee that secrets (secret encryption keys, typically) cannot ever be retrieved from the service like a real HSM would, but it would have many of the benefits as described above.
However, do note that applying appropriate protection is the key there - and that's not straightforward at all. One approach that you can take is model your potential attackers, list ways (attack paths) for compromising different aspects of different components, and then design protections against those, as long as it makes sense financially.

How to prevent direct access to cert files when connecting MQTT client with Python

I am using the pho MQTT client library successfully to connect to AWS. After the mqtt client is created, providing the necessary keys and certificates is done with a call to client.tls_set() This method requires file paths to root certificate, own certificate and private key file.
All is well and life is good except that I now need to provide this code to external contractors whom should not have direct access to these cert and key files. The contractors have a mix of PC and macOS systems. On macOS we have keychain I am familiar with but do not know how to approach this with python - examples/library references would be great. On the PC I have no idea which is the prevalent mechanism to solve this.
To add to this, I have no control over the contractor PCs/Macs - i.e., I have no ability to revoke an item in their keychain. How do I solve this?
Sorry for being such a noob in security aspects. No need to provide complete examples, just references to articles to read, courses to follow and keywords to search would be great - though code examples will be happily accepted also of course.

Short answer: you don't.
Longer answer:
If you want them to be able connect then you have no choice but to give them the cert/private key that identifies that device/user.
The control you have is issue each contractor with their own unique key/cert and if you believe key/cert has been miss used, revoke the cert at the CA and have the broker check the revocation list.
You can protect the private key with a password, but again you have to either include this in the code or give it to the contractor.
Even if the contractors were using a device with a hardware keystore (secure element) that you could securely store the private key in, all that would do is stop the user from extracting the key and moving it to a different machine, they would still be able to make use of the private key for what ever they want on that machine.
The best mitigation is to make sure the certificate has a short life and control renewing the certificate, this means if a certificate is leaked then it will stop working quickly even if you don't notice and explicitly revoke it.

How to make sure that my AJAX requests are originating from the same server in Python

I have already asked a question about IP Authentication here: TastyPie Authentication from the same server
However, I need something more! An IP address could be very easily spoofed.
Scenario: My API (TastyPie) and Client App (in javascript) are on the same server/site/domain. My users don't login. I want to consume my API in my javascript client side.
Question: How can I make sure (authentication) that my AJAX requests are originating from the same server?
I'm using Tatypie. I need to authentication that the requests from the client are being made on the same server/domain etc. I cannot use 'logged in sessions' as my users don't login.
I have looked at private keys and generating a signature but they can viewed in the javascript making that method insecure. If I do it in a way to request a signature form the server (hiding the private key in some python code) anyone can make the same http request to get_signature that my javascript makes, thus defeating the point.
I also tried to have the Django view put the signature in the view eliminating the need to make the get_signature call. This is safe, but means that I have to now refresh the page every time to get a new signature. From a users point of view only the first call to the API would work, after which they need to refresh, again pointless.
I cannot believe I'm the only person with this requirement. This is a common scenario I'm sure. Please help :) An example using custom authentication in Tastypie would be welcome too.
Thanks
Added:

Depending on your infrastructure #dragonx's answer might interest you most.
my 2c
You want to make sure that only if a client visits your website can use the api? Hmm does the bot, robot, crawler fall in the same category with the client then? Or am I wrong? This can be easily exploited in case you really want to secure it really.
I cannot believe I'm the only person with this requirement.
Maybe not, but as you can see you are prone to several attacks to your API and that can be a reason for someone not sharing your design and making security stricter with auth.
EDIT
Since we are talking about AJAX requests what does the IP part has to do with this? The IP will always be the Client's IP! So probably, you want a public API...
I would Go with the tokens/session/cookie part.
I 'd go with a generated token that lasts a little while and a flow described below.
I'd go with a limiter per some time, like Github does. Eg 60 requests per hour per ip or more for registered users
To overcome the problem with the refreshing token I would just do this:
Client visits the site
-> server generates API TOKEN INIT
-> Client gets API TOKEN INIT which is valid only for starting 1 request.
Client makes AJAX Request to API
-> Client uses API TOKEN INIT
-> Server checks against API TOKEN INIT and limits
-> Server accepts request
-> Server passes back API TOKEN
-> Client consumes response data and stores API TOKEN for further usage (Will be stored in browser memory via JS)
Client Starts Comm with the API for a limited amount of time or requests. Notice that you know also the init token date so you can use it to check against the 1st visit on the page.
The 1st token is generated via the server when the client visits.
Then the client uses that token in order to obtain a real one, that lasts for some time or something else as of limitation.
This makes someone actually visit the webpage and then he can access the API for a limit amount of time, requests perhaps etc.
This way you don't need refreshing.
Of course the above scenario could be simplified with only one token and a time limit as mentioned above.
Of course the above scenario is prone to advanced crawlers, etc since you have no authentication.
Of course a clever attacker can grab tokens from server and repeat the steps but, then you already had that that problem from start.
Some extra points
As the comments provided please close writes to the API. You don't want to be a victim of DOS attacks with writes if you have doubts about your implementation(if not use auth) or for extra security
The token scenario as described above can also become more complicated eg by constantly exchanging tokens
Just for reference GAE Cloud storage uses signed_urls for kind of the same purpose.
Hope it helps.
PS. regarding IP spoofing and Defense against spoofing attacks wikipedia says so packet's won't be returned to the attacker:
Some upper layer protocols provide their own defense against IP
spoofing attacks. For example, Transmission Control Protocol (TCP)
uses sequence numbers negotiated with the remote machine to ensure
that arriving packets are part of an established connection. Since the
attacker normally can't see any reply packets, the sequence number
must be guessed in order to hijack the connection. The poor
implementation in many older operating systems and network devices,
however, means that TCP sequence numbers can be predicted.

If it's purely the same server, you can verify requests against 127.0.0.1 or localhost.
Otherwise the solution is probably at the network level, to have a separate private subnet that you can check against. It should be difficult for an attacker to spoof your subnet without being on your subnet.

I guess you're a bit confused (or I am, please correct me). That your JS code is published on the same server as your API does not mean AJAX requests will come from your server. The clients download the JS from your server and execute it, which results in requests to your API sent from the clients, not from the same server.
Now if the above scenario correctly describes your case, what you are probably trying to do is to protect your API from bot scraping. The easiest protection is CAPTCHA, and you can find some more ideas on the Wiki page.
If you are concerned that other sites may make AJAX calls to your API to copy your site functionality, you shouldn't be--AJAX requests can only be sent to the same server as the page the JS is running on, unless it is JSONP.

Short answer: It is not possible to prevent a dedicated attacker.
You have no method of identifying a client other than with the information that they give you. For instance, username/password authentication works under the assumption that only a valid client would be able to provide valid credentials. When someone logs in, all you know is that some person provided those credentials -- you assume that this means that this means that they are a legitimate user.
Let's take a look at your scenario here, as I understand it. The only method you have of authenticating a client is IP Address, a very weak form of authentication. As you stated, this can be easily spoofed, and in with some effort your server's response can be received back to the attacker's original IP address. If this happens, you can't do anything about it. The fact is, if you assume someone from a valid IP address is a valid user, then spoofers and legitimate users are indistinguishable. This is just like if someone steals your password and tries to log in to StackOverflow. To StackOverflow, the attacker and you are indistinguishable, since all they have to go on is the username and password.
You can do fancy things with the client as mentioned in other answers, such as tokens, time limits, etc., but an dedicated attacker would be able to mimic the actions of a legitimate client, and you wouldn't be able to tell them apart because they would both appear to be from valid IP addresses. For instance, in your last example, if I was an attacker looking to make API calls, I would spoof a legitimate IP address, get the signature, and use it to make an API call, just as a legitimate client would.
If your application is critical enough to deem this level of thought into security, you should at least think of implementing something like API tokens, public key encryption, or other authentication methods that are more secure than IP addresses to tell your clients apart from any attackers. Authentication by IP address (or other easily forged tokens like hostname or headers) simply won't cut it.

may be you could achieve this by using Same-origin policy
refer http://en.wikipedia.org/wiki/Same_origin_policy

As suggested by Venkatesh Bachu, Same Origin Policy and http://en.wikipedia.org/wiki/Cross-Origin_Resource_Sharing (CORS) could be used as a solution.
In your API, you can check Origin header and respond accordingly.
Need to check if Origin header can be modified by using extensions like tamper data.
A determined hacker can still snoop by pointing browser to a local proxy server.

If this app server is running on an ordinary web server that has configurable listening IP address, set it to 127.0.0.1. With the TCPServer module, it's like
SocketServer.TCPServer(("127.0.0.1", 12345), TheHandlerClass)
Use netstat command to verify the listening address is correct as "127.0.0.1"
tcp4 0 0 127.0.0.1.12345 *.* LISTEN
This will effectively making any connection originated outside the same host impossible on the TCP level.

There are two general solution types: in-band solutions using normal web server/client mechanisms, that are easy to implement but have limitations; and out-of-band solutions that rely on you to configure something externally, that take a little more work but don't have the same limitations as in-band.
If you prefer an in-band solution, then the typical approach used to prevent cross-site request forgery (XSRF) would work well. Server issues a token with a limited life span; client uses the token in requests; privacy of token is (sort of) assured by using an HTTPS connection. This approach is used widely, and works well unless you are worried about man-in-the-middle attacks that could intercept the token, or buggy browsers that could leak data to other client-side code that's being naughty.
You can eliminate those limitations, if you're motivated, by introducing client certificates. These are kind of the flip side to the SSL certificates we all use on web servers -- they operate the same way, but are used to identify the client rather than the server. Because the certificate itself never goes over the wire (you install it locally in the browser or other client), you don't have the same threats from man-in-the-middle and browser leakage. This solution isn't used much in the wild because it's confusing to set up (very confusing for the typical user), but if you have a limited number of clients and they are under your control, then it could be feasible to deploy and manage this limited number of client certificates. The certificate operations are handled by the browser, not in client code (i.e. not in JavaScript) so your concern about key data being visible in JavaScript would not apply in this scenario.
Lastly, if you want to skip over the client configuration nonsense, use the ultimate out-of-band solution -- iptables or a similar tool to create an application-level firewall that only allows sessions that originate from network interfaces (like local loopback) that you know for certain can't be accessed off the box.

Secure credential storage in python

The attack
One possible threat model, in the context of credential storage, is an attacker which has the ability to :
inspect any (user) process memory
read local (user) files
AFAIK, the consensus on this type of attack is that it's impossible to prevent (since the credentials must be stored in memory for the program to actually use them), but there's a couple of techniques to mitigate it:
minimize the amount of time the sensitive data is stored in memory
overwrite the memory as soon as the data is not needed anymore
mangle the data in memory, keep moving it, and other security through obscurity measures
Python in particular
The first technique is easy enough to implement, possibly through a keyring (hopefully kernel space storage)
The second one is not achievable at all without writing a C module, to the best of my knowledge (but I'd love to be proved wrong here, or to have a list of existing modules)
The third one is tricky.
In particular, python being a language with very powerful introspection and reflection capabilities, it's difficult to prevent access to the credentials to anyone which can execute python code in the interpreter process.
There seems to be a consensus that there's no way to enforce private attributes and that attempts at it will at best annoy other programmers who are using your code.
The question
Taking all this into consideration, how does one securely store authentication credentials using python? What are the best practices? Can something be done about the language "everything is public" philosophy? I know "we're all consenting adults here", but should we be forced to choose between sharing our passwords with an attacker and using another language?

There are two very different reasons why you might store authentication credentials:
To authenticate your user: For example, you only allow the user access to the services after the user authenticates to your program
To authenticate the program with another program or service: For example, the user starts your program which then accesses the user's email over the Internet using IMAP.
In the first case, you should never store the password (or an encrypted version of the password). Instead, you should hash the password with a high-quality salt and ensure that the hashing algorithm you use is computationally expensive (to prevent dictionary attacks) such as PBKDF2 or bcrypt. See Salted Password Hashing - Doing it Right for many more details. If you follow this approach, even if the hacker retrieves the salted, slow-hashed token, they can't do very much with it.
In the second case, there are a number of things done to make secret discovery harder (as you outline in your question), such as:
Keeping secrets encrypted until needed, decrypting on demand, then re-encrypting immediately after
Using address space randomization so each time the application runs, the keys are stored at a different address
Using the OS keystores
Using a "hard" language such as C/C++ rather than a VM-based, introspective language such as Java or Python
Such approaches are certainly better than nothing, but a skilled hacker will break it sooner or later.
Tokens
From a theoretical perspective, authentication is the act of proving that the person challenged is who they say they are. Traditionally, this is achieved with a shared secret (the password), but there are other ways to prove yourself, including:
Out-of-band authentication. For example, where I live, when I try to log into my internet bank, I receive a one-time password (OTP) as a SMS on my phone. In this method, I prove I am by virtue of owning a specific telephone number
Security token: To log in to a service, I have to press a button on my token to get a OTP which I then use as my password.
Other devices:
SmartCard, in particular as used by the US DoD where it is called the CAC. Python has a module called pyscard to interface to this
NFC device
And a more complete list here
The commonality between all these approaches is that the end-user controls these devices and the secrets never actually leave the token/card/phone, and certainly are never stored in your program. This makes them much more secure.
Session stealing
However (there is always a however):
Let us suppose you manage to secure the login so the hacker cannot access the security tokens. Now your application is happily interacting with the secured service. Unfortunately, if the hacker can run arbitrary executables on your computer, the hacker can hijack your session for example by injecting additional commands into your valid use of the service. In other words, while you have protected the password, it's entirely irrelevant because the hacker still gains access to the 'secured' resource.
This is a very real threat, as the multiple cross-site scripting attacks have shows (one example is U.S. Bank and Bank of America Websites Vulnerable, but there are countless more).
Secure proxy
As discussed above, there is a fundamental issue in keeping the credentials of an account on a third-party service or system so that the application can log onto it, especially if the only log-on approach is a username and password.
One way to partially mitigate this by delegating the communication to the service to a secure proxy, and develop a secure sign-on approach between the application and proxy. In this approach
The application uses a PKI scheme or two-factor authentication to sign onto the secure proxy
The user adds security credentials to the third-party system to the secure proxy. The credentials are never stored in the application
Later, when the application needs to access the third-party system, it sends a request to the proxy. The proxy logs on using the security credentials and makes the request, returning results to the application.
The disadvantages to this approach are:
The user may not want to trust the secure proxy with the storage of the credentials
The user may not trust the secure proxy with the data flowing through it to the third-party application
The application owner has additional infrastructure and hosting costs for running the proxy
Some answers
So, on to specific answers:
How does one securely store authentication credentials using python?
If storing a password for the application to authenticate the user, use a PBKDF2 algorithm, such as https://www.dlitz.net/software/python-pbkdf2/
If storing a password/security token to access another service, then there is no absolutely secure way.
However, consider switching authentication strategies to, for example the smartcard, using, eg, pyscard. You can use smartcards to both authenticate a user to the application, and also securely authenticate the application to another service with X.509 certs.
Can something be done about the language "everything is public" philosophy? I know "we're all consenting adults here", but should we be forced to choose between sharing our passwords with an attacker and using another language?
IMHO there is nothing wrong with writing a specific module in Python that does it's damnedest to hide the secret information, making it a right bugger for others to reuse (annoying other programmers is its purpose). You could even code large portions in C and link to it. However, don't do this for other modules for obvious reasons.
Ultimately, though, if the hacker has control over the computer, there is no privacy on the computer at all. Theoretical worst-case is that your program is running in a VM, and the hacker has complete access to all memory on the computer, including the BIOS and graphics card, and can step your application though authentication to discover its secrets.
Given no absolute privacy, the rest is just obfuscation, and the level of protection is simply how hard it is obfuscated vs. how much a skilled hacker wants the information. And we all know how that ends, even for custom hardware and billion-dollar products.
Using Python keyring
While this will quite securely manage the key with respect to other applications, all Python applications share access to the tokens. This is not in the slightest bit secure to the type of attack you are worried about.

I'm no expert in this field and am really just looking to solve the same problem that you are, but it looks like something like Hashicorp's Vault might be able to help out quite nicely.
In particular WRT to the problem of storing credentials for 3rd part services. e.g.:
In the modern world of API-driven everything, many systems also support programmatic creation of access credentials. Vault takes advantage of this support through a feature called dynamic secrets: secrets that are generated on-demand, and also support automatic revocation.
For Vault 0.1, Vault supports dynamically generating AWS, SQL, and Consul credentials.
More links:
Github
Vault Website
Use Cases

Secure use of GAE application namespace

I'd like to have a mapping of users to accounts, and then have users directed to a namespace corresponding to their account.
Having looked at the appengine_config.py from the suggested example, there appear to be a few suggested ways to determine what the namespace ought to be, i.e.
Server name
Google Apps Domain
Cookie
I would like to have namespaces selected based on a lookup in the datastore. i.e.
namespace = user.account.name
For some user object that is linked to an account, which account has a name field. There area few ways I've posited to accomplish this:
datastore lookup on each request
memcache lookup on each request (fallback to datastore when memcache expires)
secure cookie data
The datastore lookup would be two slow. Is there any such reservation with a memcache lookup? e.g. memcache.get('nslookup:%s' % user_id), given a user_id. (I trust the users object works as expected in appengine_config.py).
Alternatively, one could use a secure cookie to solve this. I'm not satisfied with the security of the "Secure" flag (i.e. forcing SSL). However, I'm not sure about how best to secure the data in the cookie. I suppose symmetric encryption with signing with PyCrypto using a secret key in GAE along is one way to get started on this path. Although this pattern has been vetted, I'd be grateful for any thoughts on this suggested solution in particular.
Secure cookies don't seem the best route from an idealogical standpoint; I already expect to have the user identity, all I need is a mapping from the user to their account - there is no logical basis for encrypting, sending, storing, receiving, and decrypting that mapping on every request. The memcache options seems best of the three, but I'd be grateful for thoughts and input. The only reason I can think of to use secure cookies would be performance, or alternatively if memcache access were unavailable in the appengine_config.py.
Thoughts and input and challenges to my assumptions are most welcome.
Thank you for reading.
Brian

Performance-wise, anything that avoids a need for memcache or datastore lookups on each request is going to be the best option. You're confusing two definitions of 'secure' cookie, though: the 'secure' flag in the cookie spec mandates that the cookie is only sent over SSL, while in the other sense, a 'secure' cookie is one that cannot be modified undetectably by the user - which is what is most important in this use case.
There's no need to encrypt the contents, though - you want to prevent modification, not disclosure - so if you can't use an existing library, you can simply append an HMAC of the cookie to the end of it, using a secret key that you embed in your application. Verifying the HMAC on each request will be much faster than using memcache.

I think that secure cookies are the way to go because they are fast enough. A basic implementation extracted from Tornado is here (you just need the SecureCookie class and can ignore the "session" stuff):
http://code.google.com/p/webapp-improved/source/browse/extras/sessions.py#104