We Keep Coding

"/Library" directory permission denied on Mac - Python3

I'm trying to create a program that copies a directory in the library directory on mac (path : "/Library"). I use shutil which works very well in other directories but not in the Library directory...
I want to be able to compile my program, so I can't run it as root.
Here is my code :
import shutil
def copy(src_path, dir_path):
try:
shutil.copytree(src_path, dir_path)
print("Success!")
except:
print("Impossible to copy the folder...")
print("Failed!")
copy("/Users/marinnagy/Desktop/Test", "Library/Test")
I think it's because the library directory is protected and requires authentication to make changes.
Do I have to make an authentication request to the user ? Or do I need to use another method than shutil ?
Thanks for your help !

After a good deal of research and many attempts, I finally managed to copy a folder into my Library directory.
On macOS, the process of writing to a protected directory like the Library directory is blocked for python program. Once compiled (I use pyinstaller), it seems to be impossible for a python application to access this kind of folder, even if you give the app Full Disk Access in the System Preferences.
So I used some AppleScript to manage this specific copy/paste task :
on run {scr_path, dir_path} # Run with arguments
# Translate standard paths to their quoted form
set formated_scr_path to quoted form of scr_path
set formated_dir_path to quoted form of dir_path
# Run a simple shell script to copy the repertory in the other
do shell script "cp -R " & formated_scr_path & space & formated_dir_path ¬
with administrator privileges # Ask for administrator privileges
end run
Then, in my python program, I call the AppleScript program when I want to copy/past to a protected repertory like the Library repertory :
import subprocess
def copy(scr_path, dir_path):
# Use the osascript process to call the AppleScript
# Give the paths in arguments
process = subprocess.call(['osascript', "path/to/applescript",
scr_path, dir_path])
return process
copy("path/to/folder 1", "path/to/folder 2")
This method worked for me on protected repertories. The AppleScript run in the background and an authentication window pop in, asking the user to identify himself as an admin :
result screenshot

How to call salt-ssh (SSHClient) via Python API

I installed Salt in a Python 3 virtual environment and created a Salt configuration that uses a non-root folder for everything (/home/user/saltenv). When using the salt-ssh command inside the venv, e.g. salt-ssh '*' test.ping, everything works as exptected. (Please note that the config dir is resolved via a Saltfile, so the -c option is omitted, but that should not matter.)
When calling the SSHClient directly via Python however, I get no results. I already figured out that the roster file is not read, obviously resulting in an empty target list. I am stuck somehow and the documentation is not that helpful.
Here is the code:
import salt.config
from salt.client.ssh.client import SSHClient
def main():
c_path = '/home/user/saltenv/etc/salt/master'
master_opts = salt.config.client_config(c_path)
c = SSHClient(c_path=c_path, mopts=master_opts)
res = c.cmd(tgt='*', fun='test.ping')
print(res)
if __name__ == '__main__':
main()

As it seems, the processing of some options differs between the CLI and the Client. salt-ssh does not use the SSHClient. Instead, the class salt.client.ssh.SSH is used directly.
While salt-ssh adds the config_dir from the Saltfile to the opts dictionary to resolve the master config file, the SSHClient reads the config file passed to the constructor directly and config_dir is not added to the options (resulting in the roster file not being found).
My solution is to include config_dir in the master config file as well. The code from the question will then be working unchanged.
Alternative 1: If you only have one Salt configuration, it is also possible to set the environment variable SALT_CONFIG_DIR.
Alternative 2: The mopts argument of SSHClient can be used to pass a custom configuration directory, but it requires more lines of code:
config = '/home/user/saltenv/etc/salt/master'
defaults = dict(salt.config.DEFAULT_MASTER_OPTS)
defaults['config_dir'] = os.path.dirname(config)
master_opts = salt.config.client_config(config, defaults=defaults)
c = SSHClient(mopts=master_opts)

Setting GOOGLE_APPLICATION_CREDENTIALS for BigQuery Python CLI

I'm trying to connect to Google BigQuery through the BigQuery API, using Python.
I'm following this page here:
https://cloud.google.com/bigquery/bigquery-api-quickstart
My code is as follows:
import os
import argparse
from apiclient.discovery import build
from apiclient.errors import HttpError
from oauth2client.client import GoogleCredentials
GOOGLE_APPLICATION_CREDENTIALS = './Peepl-cb1dac99bdc0.json'
def main(project_id):
# Grab the application's default credentials from the environment.
credentials = GoogleCredentials.get_application_default()
print(credentials)
# Construct the service object for interacting with the BigQuery API.
bigquery_service = build('bigquery', 'v2', credentials=credentials)
try:
query_request = bigquery_service.jobs()
query_data = {
'query': (
'SELECT TOP(corpus, 10) as title, '
'COUNT(*) as unique_words '
'FROM [publicdata:samples.shakespeare];')
}
query_response = query_request.query(
projectId=project_id,
body=query_data).execute()
print('Query Results:')
for row in query_response['rows']:
print('\t'.join(field['v'] for field in row['f']))
except HttpError as err:
print('Error: {}'.format(err.content))
raise err
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('project_id', help='Your Google Cloud Project ID.')
args = parser.parse_args()
main(args.project_id)
However, when I run this code through the terminal, I get the following error:
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
As you can see in the code, I've tried to set the GOOGLE_APPLICATION_CREDENTIALS as per the link in the error. However, the error persists. Does anyone know what the issue is?
Thank you in advance.

First - Thanks for the code - this provided to be very useful.
I would also suggest adding setting the environmental variable directly in your code - as not to set it for every environment you work on.
you can use the following code:
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "path_to_your_.json_credential_file"
I found this useful when switching between different projects that require different credentials.

I'm not sure about BigQuery, but i'm using Google Data Store for saving. If you've installed gcloud sdk in your mac, you can try running this command
gcloud auth application-default login

It's looking for the environment variable in your local UNIX (or other) environment, not a variable in your python script.
You'd set that by opening up your terminal or cygwin and doing one of the following:
export GOOGLE_APPLICATION_CREDENTIALS='/path/to/your/client_secret.json'
Type that into your terminal to set the variable for just this session
Open up your .bashrc file, in UNIX by typing in nano ~/.bashrc and add this line to it underneath user specific aliases if you see that header:
GOOGLE_APPLICATION_CREDENTIALS="/full/path/to/your/client_secret.json"
Then reload it by typing source ~/.bashrc and confirm that it's set by trying echo $GOOGLE_APPLICATION_CREDENTIALS. If it returns the path, you're good.

Note: oauth2client is deprecated, instead of GoogleCredentials.get_application_default() you can use google.auth.default(). Install the package first with:
pip install google-auth
In your specific example, I see you know where the JSON file is located from your code. Instead of default credentials (from environment variables), you can use a service account directly with the google.oauth2.service_account module.
credentials = google.oauth2.service_account.Credentials.from_service_account_file(
'./Peepl-cb1dac99bdc0.json',
scopes=['https://www.googleapis.com/auth/cloud-platform'])
You can use this credentials file just as you are currently doing so by passing them to googleapiclient.discovery.build or if you are using the google-cloud-bigquery library, pass the credentials to the google.cloud.bigquery.Client constructor.

Here is a c# solution
System.Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS",#"C:\apikey.json");
string Pathsave = System.Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS");

It's looking for the environment variable. But I was able to solve this problem on Windows platform by using Application Default Credentials.
Steps that I followed:
Installed Google SDK
Then execute gcloud init steps to specify my defaults credentials and default project which you can change as and when needed. gcloud executable can be loacted in the bin directory where you chose to install Google SDK.
After successfully providing the credentials, you can check in at the location C:\Users\"yourusername"\AppData\Roaming\gcloud\legacy_credentials\"youremail"
. You can find the credentials stored in the JSON format there.
It helped me to resolve the error.

Apart from using GOOGLE_APPLICATION_CREDENTIALS (which is already described in a bunch of answers) there is one more way to set generated json credentials as a default service account:
gcloud auth activate-service-account --key-file=<path to your generated json file>
That will activate a default account (and set credentials according to the provided json file) without explicitly setting GOOGLE_APPLICATION_CREDENTIALS, and it will be still activated after re-login or reboot without modifying .bashrc.

The link provided in the error message, https://developers.google.com/identity/protocols/application-default-credentials, says to set the environment variable to point at the fail that contains the JSON service credentials. It looks like you set a Python variable. Try setting your terminal's environment variable to point at the correct file.
An alternate would be to explicitly use some other credentials when you aren't running in a GCE container, like oauth2client.client.SignedJwtAssertionCredentials and point it directly at your client secret so you don't have to indirect through an environment variable.

There is another workaround that I don't think was mentioned here yet. The google.oauth2.service_account.Credentials object offers the from_service_account_info method (see here: https://github.com/googleapis/google-auth-library-python/blob/main/google/oauth2/service_account.py).
So you can set any variable you want in your environment and read it in and pass it into the function something like this:
your_data = {
"type": os.environ.get('YOUR_ENV_VAR'),
"project_id": os.environ.get('YOUR_ENV_VAR'),
"private_key_id": os.environ.get('YOUR_ENV_VAR'),
#... and so on with all the required Google variables....
}
your_credentials = service_account.Credentials.from_service_account_info(your_data, scopes=your_scopes)
service = discovery.build(api_name, api_version, credentials=your_credentials)
I basically took all the data from my google keyfile.json and stored them in the env and did the above. That way you never need to keep your keyfile.json anywhere near your code or worse, upload it somewhere public. And that's basically it. Good luck!
PS: I forgot to mention this also, which might help someone who is running into the same issues as I did. While the above should work fine in development, in some production environments the \n will not be interpreted as a new line. Instead it will remain inside the private key. Put all of the above into a try statement and if you have the error: 'no key could be detected' then this is most likely the problem. In that case you need to replace all \\n with \n similar as to to what has been suggested by Sumit Agrawal but kind of the other way round. That's because in some environments an automatic adding of \ will occur for a new line indication such as \n in order to keep them as they are if that makes any sense. So you have to basically undo this.
You can simply do the following for one of the lines above:
"private_key": os.environ.get('YOUR_ENV_VAR').replace('\\n', '\n'),
But again try to print them to the log file / console to see how they actually look like. If you have any \n in the string you know you need to clean or convert them as explained. Good luck!

If you would like to use different credential files without setting the environmental variable, you can use the following code:
from oauth2client import service_account
from apiclient.discovery import build
import json
client_credentials = json.load(open("<path to .json credentials>"))
credentials_token = service_account._JWTAccessCredentials.from_json_keyfile_dict(client_credentials)
bigquery_service = build('bigquery', 'v2', credentials=credentials_token)
query_request = bigquery_service.jobs()
query_data = {
'query': (
'SELECT TOP(corpus, 10) as title, '
'COUNT(*) as unique_words '
'FROM [publicdata:samples.shakespeare];')
}
query_response = query_request.query(
projectId=project_id,
body=query_data).execute()
print('Query Results:')
for row in query_response['rows']:
print('\t'.join(field['v'] for field in row['f']))

Export the Google credential JSON from command line: export GOOGLE_APPLICATION_CREDENTIALS='\path\key.json'
I hope it will work fine.

You can create a client with service account credentials using from_service_account_json():
from google.cloud import bigquery
bigqueryClient = bigquery.Client.from_service_account_json('/path/to/keyfile.json')

If there is case where you can not provide credential in a file set
GOOGLE_APPLICATION_CREDENTIALS='\path\key.json'
As the service account is JSON and it contains double quote character, replace every double quote with \"
Wrap the complete JSON in double quote
Replace every \n with \\n ( on linux ) \\\n (on mac)
with above changes in service account if you export it as variable, then it should be recorded correctly.
try echo %variable_name to confirm if it looks good.

In your project folder just type:
set GOOGLE_APPLICATION_CREDENTIALS='\path\key.json'

On Windows, update the 'environmental variables for your account'.
You will most likely already have a variable called: GOOGLE_APPLICATION_CREDENTIALS
Simply update the path to /path/to/liquid-optics-xxxxxxxx.json
(you'll most likely have that file somewhere on your machine). Then refresh your environment (cmd or whatever) to pick up the change.

How to open a newtab page using the webbrowser module in python?

I'm currently trying to open my chrome default new tab page using the webbrowser module in python. I've gotten it to work for opening up random urls, however, when I try chrome://newtab as the url, I just get a message saying that there are "no apps installed to open this type of link".
Here's the relevant bit of code (not much):
import webbrowser
webbrowser.open_new_tab("chrome://newtab")
Yes, chrome is my default browser. Thanks for the help!

Notice that the documentation states that:
Note that on some platforms, trying to open a filename using this function, may work and start the operating system’s associated program. However, this is neither supported nor portable.
It has been a while since I looked at this, but my recollection is that on at least some systems, the way it works under the hood is that is passes the given URI to a system specific built-in command which then opens the URI in the system default for whatever type of URI was passed in. In other words, the default application for a given file type is used. It doesn't matter if the URI points to a local file or not. Therefore, the URI http://examplce.comn/somefile.pdf would open the PDF file on the system default PDF viewer, which may not be the browser. As the documentation notes, this works by accident due to the underlying implementation.
However, in a different OS, such a system specific command doesn't exist, and all URIs will be opened in a web browser.
You failed to mention which OS you are working on (and I forget which OS works which way), but I suspect you are working on an OS of the first type. You might (again depends on which system you have) be able override the default behavior by specifying that a specific browser be used.
You could try setting the environment variable BROWSER as an os.pathsep-separated list of browsers to try in order. Check the value of os.pathsep (import os; print os.pathsep) to see which character is used by your system (usually ':' for POSIX or ';' for Windows) and then use that character to separate items in the list. Of course, you may need to only assign one item to the list (chrome), in which case you don't need to use the separator at all. Like this (be sure to use the correct path for your system):
SET BROWSER="C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"
Or, you could try using webrowser.get() to choose your browser programically. However, support for Chrome hasn't been added until Python 3.3. If you are using Python 3.3+, then try:
import webbrowser
chrome = webbrowser.get('google-chrome') # or webbrowser.get('chrome')
chrome.open_new_tab('chrome://newtab')
Note: the above is untested. I don't know which system you have and am therefore not able to replicate your specific setup. YMMV.
Update:
As I now know you are on a pre-Python 3.3 windows machine perhaps the following will help. You can also register a browser so that Python knows about it:
pth = "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"
webbrowser.register('chrome', None, webbrowser.BackgroundBrowser(pth))
chrome = webbrowser.get('chrome')
chrome.open_new_tab('chrome://newtab')

Seemingly Bulletproof Rock Solid Command Line From Python Option
In development, the methods above worked well for me. Then I moved my code to a production server. I found different flavors of Windows don't all play the same with python's webdriver module - I was very bummed!
I needed a bulletproof method to open auto generated html reports in a specific order.
The code below is a modification of what I did that worked very well.
import subprocess as SP
import time
def display_reports_in_chrome(param_1, param_2):
front = f'start chrome.exe {param_1}\\reports\\' # beginning of command string
reports_list = [
'report_1.html', 'report_2.html', 'report_3.html', 'report_4.html']
url_list = []
for report in reports_list:
url_list.append(f'{front}{param_2}\\{report}') # complete build of commands
for url_cs in url_list: # url_cs = url open command string
time.sleep(0.1) # helps to ensure the tabs order correctly
clo = SP.run(url_cs, shell=True, capture_output=True) # actual shell command
# Stuff below makes sure it worked
check = clo.returncode == 0
error = clo.stderr.decode('utf-8')
url_open_ok = check and not error
err_msg = f'{error}. Let Thom know please!'
assert url_open_ok, err_msg
I should point out that I was enlightened to try this by this answer on SuperUser

Is there any way to modify the pydevd_file_utils.PATHS_FROM_ECLIPSE_TO_PYTHON value without having to modify that file?

I am using the pydev plugin to debug a remote application.
This (remote) application has a structure of files that differs from the structure where my Eclipse is running. This leads to problems when I set the breakpoints from the Eclipse IDE because the pydev debugger server cannot match the absolute path of the file with the file on the remote application and hence the breakpoint isn´t hit.
I dont want to hardcode the pydevd_file_utils.PATHS_FROM_ECLIPSE_TO_PYTHON constant to enable filepath translations.
Do you know some way to modify this value without changing the file?
Thanks!

There are 2 ways of setting the path translation:
Use an environment variable such as PATHS_FROM_ECLIPSE_TO_PYTHON that maps the paths from the client to the server side.
The value is a json string with a list(list(str, str)) such that:
PATHS_FROM_ECLIPSE_TO_PYTHON=[["c:/local/path", "/path/in/server"]]
Note that you may set the environment variable in any place you'd like (such as the Environment tab in the Python interpreter preferences page, in the OS itself, in the launch config, etc).
Use the pydevd API to set the tracing at runtime from the python process:
from pydevd_file_utils import setup_client_server_paths
MY_PATHS_FROM_ECLIPSE_TO_PYTHON = [
('/home/user/local-project', '/remote/path/to/project'),
]
setup_client_server_paths(MY_PATHS_FROM_ECLIPSE_TO_PYTHON)
# At this point we could connect to the remote debugger client with:
import pydevd
pydevd.settrace("10.0.0.12")
See: https://www.pydev.org/manual_adv_remote_debugger.html for more info on the Remote Debugging.
Note: the mapping set in Window > Preferences select PyDev > Debug > Source Locator doesn't really map to that environment variable nor the actual debugger mapping (that's a separate translation that only translates paths which are found on Eclipse locally and it's not really passed on to the debugger to hit breakpoints remotely).

You can do that by setting a new environment variable like this:
PATHS_FROM_ECLIPSE_TO_PYTHON='[["client_src_fullpath", "remote_src_fullpath"]]'
In linux simply run that before starting the program from the command line, or set is as a global variable.
In windows you will need to set it as a global system variable.
Variable name: PATHS_FROM_ECLIPSE_TO_PYTHON
Variable value: [["client_src_path", "remote_src_path"]]
As an alternative, you can also do this in code, BUT you need to do it BEFORE you import pydevd:
import os
os.environ.setdefault("PATHS_FROM_ECLIPSE_TO_PYTHON",'[["client_src_path","remote_src_path"]]')
import pydevd
pydevd.settrace("10.0.2.2", port=5678,stdoutToServer=True, stderrToServer=True, suspend=False,patch_multiprocessing=True)
(I'm aware this is a very old question, but none of the answers were updated to the current code)

Unfortunately there is no good way to do that.
As a workaround I explicitly replaced function NormFileToServer by adding the following code at the beginning of my source file.
def SrcPathMapping(file):
eclipse_src_path = 'C:\\tmp\\workspace\\test\\Scripts\\'
server_src_path = '/home/tester/test/Scripts/'
return file.replace(eclipse_src_path, server_src_path)
import pysrc.pydevd as pydevd
pydevd.NormFileToServer = SrcPathMapping
This simplistic mapping is sufficient when all source files are located in one directory. For proper implementation of the mapping function check NormFileToServer in the pydevd_file_utils.