We Keep Coding

Getting error while running the configuring MSTICPY on Azure sentinel notebook

I am using Azure sentinel notebook for threat intelligence. While trying to configure msticpy for it to connect to Azure sentinel but getting 'Value error'. Following is the code that I am using :
from msticpy.config import MpConfigEdit
import os
mp_conf = "msticpyconfig.yaml"
# check if MSTICPYCONFIG is already an env variable
mp_env = os.environ.get("MSTICPYCONFIG")
mp_conf = mp_env if mp_env and Path (mp_env).is_file() else mp_conf
if not Path(mp_conf).is_file():
print(
"No msticpyconfig.yaml was found!",
"Please check that there is a config.json file in your workspace folder.",
"If this is not there, go back to the Microsoft Sentinel portal and launch",
"this notebook from there.",
sep="\n"
)
else:
mpedit = MpConfigEdit(mp_conf)
mpedit.set_tab("AzureSentinel")
display(mpedit)
ValueError: File not found: 'None'.

In the Azure ML terminal, create the nbuser_settings.py file in the root of your user folder, which is the folder with your username.
In the nbuser_settings.py file, add the following lines:
import os
os.environ["MSTICPYCONFIG"] = "~/msticpyconfig.yaml"
https://learn.microsoft.com/en-us/Azure/sentinel/notebooks-msticpy-advanced?msclkid=e7cd84dfd05c11ecb0df15e0892300fc&tabs=azure-ml
Reference
Some elements of MSTICPy require configuration parameters. An example is the Threat Intelligence providers. Values for these and other parameters can be set in the msticpyconfig.yaml file.
The package has a default configuration file, which is stored in the package directory. You should not need to edit this file directly. Instead you can create a custom file with your own parameters - these settings will combine with or override the settings in the default file.
By default, the custom msticpyconfig.yaml is read from the current directory. You can specify an explicit location using an environment variable MSTICPYCONFIG.
You should also read the MSTICPy Settings Editor document to see how to configure settings using and interactive User Interface from a Jupyter notebook.
!!! NOTE !!! For the Linux and Windows options, you'll need to restart your Jupyter server for it to pick up the environment variable that you defined.
https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html?msclkid=96fde57dd04d11ec9e5406de243d7c67

The author of msticpy has posted the issue on github & we have to wait for the latest release. Please follow the thread for more details:
https://github.com/microsoft/msticpy/issues/393

Python Updating system time

I have an application that is time sensitive to the time on a remote server - it needs to be run at a particular instant. Looping and retrieving the server time continuously does not get me as close as i'd like. I am trying to retrieve the server time and update my system's time (Python 3.7, windows 10) in advance so the module can restart as close as possible to the server's time. I have found the SetSysemTime function in win32api.
Here is the two line module i created to test various approaches:
import win32api
win32api.SetSystemTime(2020,9,1,21,9,10,10,0)
When i run this i get the following error:
Traceback (most recent call last):
File "C:\Users\pinev\AppData\Local\Programs\Python\Python37\Projects\Threading Tests\settime.py", line 4, in
win32api.SetSystemTime(2020,9,1,21,9,10,10,0)
pywintypes.error: (1314, 'SetSystemTime', 'A required privilege is not held by the client.')
So it seems i have the right solution, but somehow the module's privilege needs to be changed in order for Win10 to process the request. I have researched ways to set the privilege in a Python module and can't seem to find anything that works. If anyone could could provide a solution or a reference it would be greatly appreciated.
Thanks in advance.

If you just need to elevate privileges to run Python programs, you can launch the program as an administrator.
This is also the simplest and most direct method I think.
Test code:
import win32api
import ctypes, sys
def is_admin():
try:
return ctypes.windll.shell32.IsUserAnAdmin()
except:
return False
if is_admin():
win32api.SetSystemTime(2020,9,1,21,9,10,10,0)
else:
# Re-run the program with admin rights
ctypes.windll.shell32.ShellExecuteW(None, "runas", sys.executable, __file__, None, 1)

How to open a newtab page using the webbrowser module in python?

I'm currently trying to open my chrome default new tab page using the webbrowser module in python. I've gotten it to work for opening up random urls, however, when I try chrome://newtab as the url, I just get a message saying that there are "no apps installed to open this type of link".
Here's the relevant bit of code (not much):
import webbrowser
webbrowser.open_new_tab("chrome://newtab")
Yes, chrome is my default browser. Thanks for the help!

Notice that the documentation states that:
Note that on some platforms, trying to open a filename using this function, may work and start the operating system’s associated program. However, this is neither supported nor portable.
It has been a while since I looked at this, but my recollection is that on at least some systems, the way it works under the hood is that is passes the given URI to a system specific built-in command which then opens the URI in the system default for whatever type of URI was passed in. In other words, the default application for a given file type is used. It doesn't matter if the URI points to a local file or not. Therefore, the URI http://examplce.comn/somefile.pdf would open the PDF file on the system default PDF viewer, which may not be the browser. As the documentation notes, this works by accident due to the underlying implementation.
However, in a different OS, such a system specific command doesn't exist, and all URIs will be opened in a web browser.
You failed to mention which OS you are working on (and I forget which OS works which way), but I suspect you are working on an OS of the first type. You might (again depends on which system you have) be able override the default behavior by specifying that a specific browser be used.
You could try setting the environment variable BROWSER as an os.pathsep-separated list of browsers to try in order. Check the value of os.pathsep (import os; print os.pathsep) to see which character is used by your system (usually ':' for POSIX or ';' for Windows) and then use that character to separate items in the list. Of course, you may need to only assign one item to the list (chrome), in which case you don't need to use the separator at all. Like this (be sure to use the correct path for your system):
SET BROWSER="C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"
Or, you could try using webrowser.get() to choose your browser programically. However, support for Chrome hasn't been added until Python 3.3. If you are using Python 3.3+, then try:
import webbrowser
chrome = webbrowser.get('google-chrome') # or webbrowser.get('chrome')
chrome.open_new_tab('chrome://newtab')
Note: the above is untested. I don't know which system you have and am therefore not able to replicate your specific setup. YMMV.
Update:
As I now know you are on a pre-Python 3.3 windows machine perhaps the following will help. You can also register a browser so that Python knows about it:
pth = "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"
webbrowser.register('chrome', None, webbrowser.BackgroundBrowser(pth))
chrome = webbrowser.get('chrome')
chrome.open_new_tab('chrome://newtab')

Seemingly Bulletproof Rock Solid Command Line From Python Option
In development, the methods above worked well for me. Then I moved my code to a production server. I found different flavors of Windows don't all play the same with python's webdriver module - I was very bummed!
I needed a bulletproof method to open auto generated html reports in a specific order.
The code below is a modification of what I did that worked very well.
import subprocess as SP
import time
def display_reports_in_chrome(param_1, param_2):
front = f'start chrome.exe {param_1}\\reports\\' # beginning of command string
reports_list = [
'report_1.html', 'report_2.html', 'report_3.html', 'report_4.html']
url_list = []
for report in reports_list:
url_list.append(f'{front}{param_2}\\{report}') # complete build of commands
for url_cs in url_list: # url_cs = url open command string
time.sleep(0.1) # helps to ensure the tabs order correctly
clo = SP.run(url_cs, shell=True, capture_output=True) # actual shell command
# Stuff below makes sure it worked
check = clo.returncode == 0
error = clo.stderr.decode('utf-8')
url_open_ok = check and not error
err_msg = f'{error}. Let Thom know please!'
assert url_open_ok, err_msg
I should point out that I was enlightened to try this by this answer on SuperUser

How to secure a username and password in the python module? [duplicate]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I'm writing a small Python script which will periodically pull information from a 3rd party service using a username and password combo. I don't need to create something that is 100% bulletproof (does 100% even exist?), but I would like to involve a good measure of security so at the very least it would take a long time for someone to break it.
This script won't have a GUI and will be run periodically by cron, so entering a password each time it's run to decrypt things won't really work, and I'll have to store the username and password in either an encrypted file or encrypted in a SQLite database, which would be preferable as I'll be using SQLite anyway, and I might need to edit the password at some point. In addition, I'll probably be wrapping the whole program in an EXE, as it's exclusively for Windows at this point.
How can I securely store the username and password combo to be used periodically via a cron job?

The python keyring library integrates with the CryptProtectData API on Windows (along with relevant API's on Mac and Linux) which encrypts data with the user's logon credentials.
Simple usage:
import keyring
# the service is just a namespace for your app
service_id = 'IM_YOUR_APP!'
keyring.set_password(service_id, 'dustin', 'my secret password')
password = keyring.get_password(service_id, 'dustin') # retrieve password
Usage if you want to store the username on the keyring:
import keyring
MAGIC_USERNAME_KEY = 'im_the_magic_username_key'
# the service is just a namespace for your app
service_id = 'IM_YOUR_APP!'
username = 'dustin'
# save password
keyring.set_password(service_id, username, "password")
# optionally, abuse `set_password` to save username onto keyring
# we're just using some known magic string in the username field
keyring.set_password(service_id, MAGIC_USERNAME_KEY, username)
Later to get your info from the keyring
# again, abusing `get_password` to get the username.
# after all, the keyring is just a key-value store
username = keyring.get_password(service_id, MAGIC_USERNAME_KEY)
password = keyring.get_password(service_id, username)
Items are encrypted with the user's operating system credentials, thus other applications running in your user account would be able to access the password.
To obscure that vulnerability a bit you could encrypt/obfuscate the password in some manner before storing it on the keyring. Of course, anyone who was targeting your script would just be able to look at the source and figure out how to unencrypt/unobfuscate the password, but you'd at least prevent some application vacuuming up all passwords in the vault and getting yours as well.

There are a few options for storing passwords and other secrets that a Python program needs to use, particularly a program that needs to run in the background where it can't just ask the user to type in the password.
Problems to avoid:
Checking the password in to source control where other developers or even the public can see it.
Other users on the same server reading the password from a configuration file or source code.
Having the password in a source file where others can see it over your shoulder while you are editing it.
Option 1: SSH
This isn't always an option, but it's probably the best. Your private key is never transmitted over the network, SSH just runs mathematical calculations to prove that you have the right key.
In order to make it work, you need the following:
The database or whatever you are accessing needs to be accessible by SSH. Try searching for "SSH" plus whatever service you are accessing. For example, "ssh postgresql". If this isn't a feature on your database, move on to the next option.
Create an account to run the service that will make calls to the database, and generate an SSH key.
Either add the public key to the service you're going to call, or create a local account on that server, and install the public key there.
Option 2: Environment Variables
This one is the simplest, so it might be a good place to start. It's described well in the Twelve Factor App. The basic idea is that your source code just pulls the password or other secrets from environment variables, and then you configure those environment variables on each system where you run the program. It might also be a nice touch if you use default values that will work for most developers. You have to balance that against making your software "secure by default".
Here's an example that pulls the server, user name, and password from environment variables.
import os
server = os.getenv('MY_APP_DB_SERVER', 'localhost')
user = os.getenv('MY_APP_DB_USER', 'myapp')
password = os.getenv('MY_APP_DB_PASSWORD', '')
db_connect(server, user, password)
Look up how to set environment variables in your operating system, and consider running the service under its own account. That way you don't have sensitive data in environment variables when you run programs in your own account. When you do set up those environment variables, take extra care that other users can't read them. Check file permissions, for example. Of course any users with root permission will be able to read them, but that can't be helped. If you're using systemd, look at the service unit, and be careful to use EnvironmentFile instead of Environment for any secrets. Environment values can be viewed by any user with systemctl show.
Option 3: Configuration Files
This is very similar to the environment variables, but you read the secrets from a text file. I still find the environment variables more flexible for things like deployment tools and continuous integration servers. If you decide to use a configuration file, Python supports several formats in the standard library, like JSON, INI, netrc, and XML. You can also find external packages like PyYAML and TOML. Personally, I find JSON and YAML the simplest to use, and YAML allows comments.
Three things to consider with configuration files:
Where is the file? Maybe a default location like ~/.my_app, and a command-line option to use a different location.
Make sure other users can't read the file.
Obviously, don't commit the configuration file to source code. You might want to commit a template that users can copy to their home directory.
Option 4: Python Module
Some projects just put their secrets right into a Python module.
# settings.py
db_server = 'dbhost1'
db_user = 'my_app'
db_password = 'correcthorsebatterystaple'
Then import that module to get the values.
# my_app.py
from settings import db_server, db_user, db_password
db_connect(db_server, db_user, db_password)
One project that uses this technique is Django. Obviously, you shouldn't commit settings.py to source control, although you might want to commit a file called settings_template.py that users can copy and modify.
I see a few problems with this technique:
Developers might accidentally commit the file to source control. Adding it to .gitignore reduces that risk.
Some of your code is not under source control. If you're disciplined and only put strings and numbers in here, that won't be a problem. If you start writing logging filter classes in here, stop!
If your project already uses this technique, it's easy to transition to environment variables. Just move all the setting values to environment variables, and change the Python module to read from those environment variables.

After looking though the answers to this and related questions, I've put together some code using a few of the suggested methods for encrypting and obscuring secret data. This code is specifically for when the script has to run without user intervention (if the user starts it manually, it's best to have them put in the password and only keep it in memory as the answer to this question suggests). This method isn't super-secure; fundamentally, the script can access the secret info so anyone who has full system access has the script and its associated files and can access them. What this does do id obscures the data from casual inspection and leaves the data files themselves secure if they are examined individually, or together without the script.
My motivation for this is a project that polls some of my bank accounts to monitor transactions - I need it to run in the background without me re-entering passwords every minute or two.
Just paste this code at the top of your script, change the saltSeed and then use store() retrieve() and require() in your code as needed:
from getpass import getpass
from pbkdf2 import PBKDF2
from Crypto.Cipher import AES
import os
import base64
import pickle
### Settings ###
saltSeed = 'mkhgts465wef4fwtdd' # MAKE THIS YOUR OWN RANDOM STRING
PASSPHRASE_FILE = './secret.p'
SECRETSDB_FILE = './secrets'
PASSPHRASE_SIZE = 64 # 512-bit passphrase
KEY_SIZE = 32 # 256-bit key
BLOCK_SIZE = 16 # 16-bit blocks
IV_SIZE = 16 # 128-bits to initialise
SALT_SIZE = 8 # 64-bits of salt
### System Functions ###
def getSaltForKey(key):
return PBKDF2(key, saltSeed).read(SALT_SIZE) # Salt is generated as the hash of the key with it's own salt acting like a seed value
def encrypt(plaintext, salt):
''' Pad plaintext, then encrypt it with a new, randomly initialised cipher. Will not preserve trailing whitespace in plaintext!'''
# Initialise Cipher Randomly
initVector = os.urandom(IV_SIZE)
# Prepare cipher key:
key = PBKDF2(passphrase, salt).read(KEY_SIZE)
cipher = AES.new(key, AES.MODE_CBC, initVector) # Create cipher
return initVector + cipher.encrypt(plaintext + ' '*(BLOCK_SIZE - (len(plaintext) % BLOCK_SIZE))) # Pad and encrypt
def decrypt(ciphertext, salt):
''' Reconstruct the cipher object and decrypt. Will not preserve trailing whitespace in the retrieved value!'''
# Prepare cipher key:
key = PBKDF2(passphrase, salt).read(KEY_SIZE)
# Extract IV:
initVector = ciphertext[:IV_SIZE]
ciphertext = ciphertext[IV_SIZE:]
cipher = AES.new(key, AES.MODE_CBC, initVector) # Reconstruct cipher (IV isn't needed for edecryption so is set to zeros)
return cipher.decrypt(ciphertext).rstrip(' ') # Decrypt and depad
### User Functions ###
def store(key, value):
''' Sore key-value pair safely and save to disk.'''
global db
db[key] = encrypt(value, getSaltForKey(key))
with open(SECRETSDB_FILE, 'w') as f:
pickle.dump(db, f)
def retrieve(key):
''' Fetch key-value pair.'''
return decrypt(db[key], getSaltForKey(key))
def require(key):
''' Test if key is stored, if not, prompt the user for it while hiding their input from shoulder-surfers.'''
if not key in db: store(key, getpass('Please enter a value for "%s":' % key))
### Setup ###
# Aquire passphrase:
try:
with open(PASSPHRASE_FILE) as f:
passphrase = f.read()
if len(passphrase) == 0: raise IOError
except IOError:
with open(PASSPHRASE_FILE, 'w') as f:
passphrase = os.urandom(PASSPHRASE_SIZE) # Random passphrase
f.write(base64.b64encode(passphrase))
try: os.remove(SECRETSDB_FILE) # If the passphrase has to be regenerated, then the old secrets file is irretrievable and should be removed
except: pass
else:
passphrase = base64.b64decode(passphrase) # Decode if loaded from already extant file
# Load or create secrets database:
try:
with open(SECRETSDB_FILE) as f:
db = pickle.load(f)
if db == {}: raise IOError
except (IOError, EOFError):
db = {}
with open(SECRETSDB_FILE, 'w') as f:
pickle.dump(db, f)
### Test (put your code here) ###
require('id')
require('password1')
require('password2')
print
print 'Stored Data:'
for key in db:
print key, retrieve(key) # decode values on demand to avoid exposing the whole database in memory
# DO STUFF
The security of this method would be significantly improved if os permissions were set on the secret files to only allow the script itself to read them, and if the script itself was compiled and marked as executable only (not readable). Some of that could be automated, but I haven't bothered. It would probably require setting up a user for the script and running the script as that user (and setting ownership of the script's files to that user).
I'd love any suggestions, criticisms or other points of vulnerability that anyone can think of. I'm pretty new to writing crypto code so what I've done could almost certainly be improved.

I recommend a strategy similar to ssh-agent. If you can't use ssh-agent directly you could implement something like it, so that your password is only kept in RAM. The cron job could have configured credentials to get the actual password from the agent each time it runs, use it once, and de-reference it immediately using the del statement.
The administrator still has to enter the password to start ssh-agent, at boot-time or whatever, but this is a reasonable compromise that avoids having a plain-text password stored anywhere on disk.

There's not much point trying to encrypt the password: the person you're trying to hide it from has the Python script, which will have the code to decrypt it. The fastest way to get the password will be to add a print statement to the Python script just before it uses the password with the third-party service.
So store the password as a string in the script, and base64 encode it so that just reading the file isn't enough, then call it a day.

I think the best you can do is protect the script file and system it's running on.
Basically do the following:
Use file system permissions (chmod 400)
Strong password for owner's account on the system
Reduce ability for system to be compromised (firewall, disable unneeded services, etc)
Remove administrative/root/sudo privileges for those that do not need it

I used Cryptography because I had troubles installing (compiling) other commonly mentioned libraries on my system. (Win7 x64, Python 3.5)
from cryptography.fernet import Fernet
key = Fernet.generate_key()
cipher_suite = Fernet(key)
cipher_text = cipher_suite.encrypt(b"password = scarybunny")
plain_text = cipher_suite.decrypt(cipher_text)
My script is running in a physically secure system/room. I encrypt credentials with an "encrypter script" to a config file. And then decrypt when I need to use them.
"Encrypter script" is not on the real system, only encrypted config file is. Someone who analyses the code can easily break the encryption by analysing the code, but you can still compile it into an EXE if necessary.

operating systems often have support for securing data for the user. in the case of windows it looks like it's http://msdn.microsoft.com/en-us/library/aa380261.aspx
you can call win32 apis from python using http://vermeulen.ca/python-win32api.html
as far as i understand, this will store the data so that it can be accessed only from the account used to store it. if you want to edit the data you can do so by writing code to extract, change and save the value.

What is the Windows equivalent of pwd.getpwnam(username).pw_dir?

The Python pwd module provides access to getpwnam(3) POSIX API, which can be used to get the home directory for a particular user by username, as well determining if the username is valid at all. pwd.getpwnam will raise an exception if called with a non-existent username.
At first it seems like the same result can be achieved in a cross-platform manner via os.path.expanduser('~username'). However, it appears that with Python 2.6 on Windows XP this won't actually produce a failure for a non-existent username. Furthermore, on Python 2.5 on Windows XP, it seems to fail even for valid users.
Can this information be obtained reliably on Windows? How?

Reading the 2.6 documentation shows that os.path.expanduser() is broken on Windows:
On Windows, HOME and USERPROFILE will
be used if set, otherwise a
combination of HOMEPATH and HOMEDRIVE
will be used. An initial ~user is
handled by stripping the last
directory component from the created
user path derived above.
Say whaat? This assumes all user homes have to be under the same parent directory. Nuh-ugh!
It was a bit hard to dig but here is a solution that will look up a local user by given name:
from win32security import LookupAccountName, ConvertSidToStringSid
from _winreg import OpenKey, QueryValueEx, HKEY_LOCAL_MACHINE
def getUserDir(userName):
ssid = ConvertSidToStringSid(LookupAccountName(None, userName)[0])
key = OpenKey(HKEY_LOCAL_MACHINE, r'SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\\' + ssid)
return QueryValueEx(key, 'ProfileImagePath')[0]

I am new to Windows security... but reading MSDN and some blogs it seems to me that the way MS want us to handle other users specific data is by getting a user token.
There used to be a nice wiki of Keith Brown .Net Developers Guide to Windows Security... you can still find it in Google cache for "pluralsight keith.guidebook"
Case 1: If you don't have the user password:
For local accounts you can try reading the Windows registry as Nas Banov already suggested and there are some other recipes on SO or the Internet.
I am not sure how various Windows versions behaves for freshly create users ... those which have never performed an interactive session login ... does it automatically creates their registry, home folder and profile data?
I have done some tests on Windows XP and those registry keys were not present after creating an local account ... but in this case you can try to guess it based in All Users registry values ... or just fail :)
For desktop applications, when the application is running as a logged in user, I am using something like this to get the home folder.... and to get the equivalent of ~/.local I am using CSIDL_APPDATA, for roaming profiles, or just CSIDL_LOCAL_APPDATA.
from win32com.shell import shell, shellcon
# See microsoft references for further CSIDL constants
# http://msdn.microsoft.com/en-us/library/bb762181(VS.85).aspx
folder_name = shell.SHGetFolderPath(0, shellcon.CSIDL_PROFILE, 0, 0)
Reading Keith Brown article "How To Get A Token For A User" .. you can look for some other ways of getting an user token without a password...
Case 2: If you have the user password:
Reading the MSDN I got the impressing that if I have an user token, I can get its folders by calling something like the code below... but it did not worked for me. (not sure why)
token = win32security.LogonUser(
username,
None, # we uses UPN format for username
password,
win32security.LOGON32_LOGON_NETWORK,
win32security.LOGON32_PROVIDER_DEFAULT,
)
folder_name = shell.SHGetFolderPath(0, shellcon.CSIDL_PROFILE, token, 0)
This is why I ended up with this code...which is far from being perfect due to the fact that it requires username and password.
token = win32security.LogonUser(
username,
None, # Here should be the domain ... or just go with default values
password,
win32security.LOGON32_LOGON_NETWORK,
win32security.LOGON32_PROVIDER_DEFAULT,
)
win32security.ImpersonateLoggedOnUser(token)
folder_name = shell.SHGetFolderPath(0, shellcon.CSIDL_PROFILE, 0, 0)
win32security.RevertToSelf()
This question is somehow related: How to find the real user home directory using python?

you could go the win32api.GetUserName() (current user only) or win32net.NetUserGetInfo() (any user on any server, localhost included) route. the latter could be a bit slow since it can take some time to get this information back from the OS.
import win32net
def userDir(username):
return win32net.NetUserGetInfo(None, username, 1).get("home_dir")
alternatively you could expand the environment variable USERPROFILE on windows or HOME on unix to get the information about the currently logged in user:
def userDir():
if os.platform.system() == 'Windows':
return os.environ['USERPROFILE']
elif os.platform.system() == 'Linux':
return os.environ['HOME']
else:
return None

This seems to be only applicable to the current user, but on my (winxp) machine, os.path.expanduser('~') returns my home directory.