I am having some issues getting gcloud to run in a Bazel genrule. Looks like python path related issues.
genrule(
name="foo",
outs=["bar"],
srcs=[":bar.enc"],
cmd="gcloud decrypt --location=global --keyring=foo --key=bar --plaintext-file $# --ciphertext-file $(location bar.enc)"
)
The exception is:
ImportError: No module named traceback
From:
try:
gcloud_main = _import_gcloud_main()
except Exception as err: # pylint: disable=broad-except
# We want to catch *everything* here to display a nice message to the user
# pylint:disable=g-import-not-at-top
import traceback
# We DON'T want to suggest `gcloud components reinstall` here (ex. as
# opposed to the similar message in gcloud_main.py), as we know that no
# commands will work.
sys.stderr.write(
('ERROR: gcloud failed to load: {0}\n{1}\n\n'
'This usually indicates corruption in your gcloud installation or '
'problems with your Python interpreter.\n\n'
'Please verify that the following is the path to a working Python 2.7 '
'executable:\n'
' {2}\n\n'
'If it is not, please set the CLOUDSDK_PYTHON environment variable to '
'point to a working Python 2.7 executable.\n\n'
'If you are still experiencing problems, please reinstall the Cloud '
'SDK using the instructions here:\n'
' https://cloud.google.com/sdk/\n').format(
err,
'\n'.join(traceback.format_exc().splitlines()[2::2]),
sys.executable))
sys.exit(1)
My questions are:
How do I best call gcloud from a genrule?
What are the parameters needed to specify the python path?
How is Bazel blocking this?
Update:
Able to get it to run by specifying the CLOUDSDK_PYTHON.
Indeed, bazel runs in a sandbox, hence gcloud cannot find its dependencies. Acutally, I'm surprised gcloud can be invoked at all.
To proceed, I would wrap gcloud in a bazel py_binary and refer it with tools attribute in the genrule. You also need to wrap it with location in the cmd. In the end, you will have
genrule(
name = "foo",
outs = ["bar"],
srcs = [":bar.enc"],
cmd = "$(location //third_party/google/gcloud) decrypt --location=global --keyring=foo --key=bar --plaintext-file $# --ciphertext-file $(location bar.enc)",
tools = ["//third_party/google/gcloud"],
)
And for that you define in third_party/google/gcloud/BUILD (or anywhere your want, I just used a path that makes sense to me)
py_binary(
name = "gcloud",
srcs = ["gcloud.py"],
main = "gcloud.py",
visibility = ["//visibility:public"],
deps = [
":gcloud_sdk",
],
)
py_library(
name = "gcloud_sdk",
srcs = glob(
["**/*.py"],
exclude = ["gcloud.py"],
# maybe exclude tests and unrelated code, too.
),
deps = [
# Whatever extra deps are needed by gcloud to compile
]
)
I had a similar issue, worked for me running this command:
export CLOUDSDK_PYTHON=/usr/bin/python
(this was answered above as an update but I felt to post the whole command for future people coming here)
Related
I was trying to run through some of the examples from pyinfra's docs to see if this tool would work for me (so I could manage some servers using good old Python) and I even found they had an example for installing virtualbox on a host machine, but I can't get even the example working. I'm getting a weird error that makes me think there's been an update to the tool since the examples, and even the docs, have been updated. Just wondering if anyone else has a way to get this to work with pyinfra (currently trying with version 1.5).
The example code:
from pyinfra import config, host
from pyinfra.facts.server import LinuxDistribution, LinuxName, OsVersion
from pyinfra.operations import apt, python, server
config.SUDO = True
virtualbox_version = '6.1'
def verify_virtualbox_version(state, host, version):
command = '/usr/bin/virtualbox --help'
status, stdout, stderr = host.run_shell_command(state, command=command, sudo=config.SUDO)
assert status is True # ensure the command executed OK
if version not in str(stdout):
raise Exception('`{}` did not work as expected.stdout:{} stderr:{}'.format(
command, stdout, stderr))
if host.get_fact(LinuxName) == 'Ubuntu':
code_name = host.get_fact(LinuxDistribution)['release_meta'].get('DISTRIB_CODENAME')
apt.packages(
name='Install packages',
packages=['wget'],
update=True,
)
apt.key(
name='Install VirtualBox key',
src='https://www.virtualbox.org/download/oracle_vbox_2016.asc',
)
apt.repo(
name='Install VirtualBox repo',
src='deb https://download.virtualbox.org/virtualbox/debian {} contrib'.format(code_name),
)
# install kernel headers
# Note: host.get_fact(OsVersion) is the same as `uname -r` (ex: '4.15.0-72-generic')
apt.packages(
{
'Install VirtualBox version {} and '
'kernel headers for {}'.format(virtualbox_version, host.get_fact(OsVersion)),
},
[
'virtualbox-{}'.format(virtualbox_version),
'linux-headers-{}'.format(host.get_fact(OsVersion)),
],
update=True,
)
server.shell(
name='Run vboxconfig which will stop/start VirtualBox services and build kernel modules',
commands='/sbin/vboxconfig',
)
python.call(
name='Verify VirtualBox version',
function=verify_virtualbox_version,
version=virtualbox_version,
)
Here is the error that I'm seeing:
File "install_virtualbox.py", line 21, in <module>
if host.get_fact(LinuxName) == 'Ubuntu':
AttributeError: module 'pyinfra.api.host' has no attribute 'get_fact'
I checked the source and I can't argue with my editor when it says it 'cannot find a reference to config' or 'init', as I don't see them either. But then how is anyone getting facts about hosts? Looks like the logic may have been moved to the 'pyinfra.api.host' and 'pyinfra.api.config' packages, but the logic there looks totally different.
Hoping I'm just missing something that someone who's been using the tool can help explain to me.
I have created the SConstruct to install the systemd user services but when I try to scons uninstall the temporary services files are creates which should not happen.
import os
PATH_WD = os.path.abspath(os.curdir)
env = Environment(
SUBSTFILESUFFIX = '.service',
SUBST_DICT = { '{{PATH_ROOT}}' : os.path.dirname(PATH_WD) },
ENV = {
'DBUS_SESSION_BUS_ADDRESS' : os.environ['DBUS_SESSION_BUS_ADDRESS'],
'XDG_RUNTIME_DIR' : os.environ['XDG_RUNTIME_DIR']
}
)
INSTALLED = env.Install(
target = os.path.expanduser('~/.config/systemd/user/'),
source = [
env.Substfile('service1'),
env.Substfile('service2'),
]
)
env.AddPostAction(INSTALLED, env.Action('systemctl --user daemon-reload'))
Alias('install', INSTALLED)
NoClean(INSTALLED)
Command('uninstall', INSTALLED, Delete(INSTALLED))
Default('install')
Second try..
Here's a trivial example which should work for you..
env=Environment()
prog=env.Program('main.c')
Default(prog)
installed_prog = env.Install('install_dir', prog)
Alias('install', installed_prog)
NoClean(installed_prog)
# You don't have to specify targets to Alias.. so it won't
# try to build those before executing the Action
Alias('uninstall', action=Delete(installed_prog))
AlwaysBuild('uninstall')
SCons builder calls are statements of relationships between nodes. You've associated the target "uninstall" with the source INSTALLED by calling the Command builder. So in order to "build" this target, you need the source, and the source is the list of nodes returned by calling the Install builder. So the Install has to happen before the uninstall can take place. Is there a reason you don't want SCons' clean functionality to be used here? To see this, try: scons --tree=all,linedraw -n uninstall
Trying to run git-filter-repo command from python. I can't figure out how this should be done.
git filter-repo --path README.md --path guides/ --path tools/releases
So far I have:
filtering_options = git_filter_repo.FilteringOptions.default_options()
filtering_options.source = _fragment_repo_git_directory(workspace, "svn-import")
filtering_options.target = _main_repo_git_directory(workspace).encode()
filtering_options.force = True
filtering_options.replace_refs = "update no add"
repo_filter = git_filter_repo.RepoFilter(
filtering_options,
# ?????????
)
repo_filter.run()
You can take inspiration from the newren/git-filter-repo/contrib/filter-repo-demos/lint-history script, which does use a callback function as your # ????????? argument:
try:
import git_filter_repo as fr
except ImportError:
raise SystemExit("Error: Couldn't find git_filter_repo.py. Did you forget to make a symlink to git-filter-repo named git_filter_repo.py or did you forget to put the latter in your PYTHONPATH?")
def lint_non_binary_blobs(blob, metadata):
...
filter = fr.RepoFilter(args, blob_callback=lint_non_binary_blobs)
filter.run()
I am a complete Python beginner and try to write a Python script to automate the setup of a SDK on Linux machines from remote Github repositories.
The script starts by performing some basic preliminary operations, especially the check/setup of several packages (git, docker, pip, etc.).
For now, I target Debian (Stretch, Buster), Centos (6, 7) and Ubuntu Server 18.04LTS.
Of course, I want the script to run on the widest range of linux machines.
Today I rely on available package managers (apt-get and yum), roughly requested through subprocess.call() statements.
I customize the related commands using nasty script configuration variables like below :
import platform
distribution = platform.dist()[0]
version = platform.dist()[1]
if distribution == 'debian':
pkgInstaller = 'dpkg'
pkmManager = 'apt-get'
checkIfInstalled = '-s'
installPackage = 'install'
yesToAll = '-y'
dockerPackage = 'docker-ce'
elif distribution == 'centos':
pkgInstaller = 'rpm'
pkgManager = 'yum'
checkIfInstalled = '-q'
installPackage = 'install'
yesToAll = '-y'
dockerPackage = 'docker'
I then simply loop on an array containing the names of packages to be installed, then run the command through subprocess.call() :
prerequisites = ['git', dockerPackage, 'doxygen', 'python2-pip']
for pkg in prerequisites:
pgkInstallation = subprocess.call(['sudo', pkgManager, yesToAll, installPackage, pkg])
While this approach may have the benefit of not having too much bonding to third-party Python modules, I guess there are... some smarter ways of doing such simple operation ?
Usually when doing switch statements like this, a dictionary might be a bit more useful. Also, normally I'm not one to try to PEP-8 things, but this is an instance where PEP-8 might really help your readability by not matching up your equals signs for all of your lines of code.
The dict will hold your distro as the key, and your vars as a value wrapped in a tuple
options = {
'debian': ('dpkg', 'apt-get', '-s', 'install', '-y', 'docker-ce'),
'centos': ('rpm', 'yum', '-q', 'install', '-y', 'docker'),
}
# unpack this function call here
distribution, version, *_ = platform.dist()
# now get the match
pkg_installer, pkg_manager, check, install_pkg, yes_to_all, docker = options[distribution]
requisites = ['git', docker, 'doxygen', 'python2-pip']
for pkg in requisites:
pgkInstallation = subprocess.call(['sudo', pkg_manager, yes_to_all, install_pkg, pkg])
The options[distribution] call will raise a KeyError for unsupported distributions, so you can probably catch that and raise something a bit more useful like:
try:
pkg_installer, pkg_manager, check, install_pkg, yes_to_all, docker = options[distribution]
except KeyError as e:
raise ValueError(f"Got unsupported OS, expected one of {', '.join(options.keys())}") from e
To make it less verbose, the only var you use out of order is docker, so you can house all of the others in a single var:
try:
*args, docker = options[distribution]
except KeyError as e:
raise ValueError(f"Got unsupported OS, expected one of {', '.join(options.keys())}") from e
requisites = ['git', docker, 'doxygen', 'python2-pip']
for pkg in requisites:
pgkInstallation = subprocess.call(['sudo', *args, pkg])
I am using jenkins rest API to recurse through jobs and then reconfigure this one. All methods work except one. He's is my code :
def get_server_instance():
jenkins_url = 'xxxx'
#server = Jenkins(jenkins_url, username = '', password = '')
# Connect to instance - username and password are optional
server = jenkins.Jenkins(jenkins_url, username = '', password = '')
return server
def get_job_details():
# Refer Example #1 for definition of function 'get_server_instance'
server = get_server_instance()
for job in server.get_jobs_list():
if job == "GithubMigration":
configuration = server.get_job(job).get_config().encode('utf-8')
#server.reconfig_job(job, configuration)
if server.has_job("GithubMigration"):
server.reconfig_job('GithubMigration', config_xml)
It gets my configuration.xml, find the job as well but fails on server.reconfig_job('GithubMigration', config_xml) with the error , AttributeError: 'Jenkins' object has no attribute 'reconfig_job'
when obviously this functions exists in the jenkins rest API and yes I'm importing jenkins, from jenkinsapi.jenkins import Jenkins .
Edit 1 - I uninstalled Jenkinsapi and have only python-jenkins module and now it fails even before saying
AttributeError: 'module' object has no attribute 'Jenkins' for line : AttributeError: 'module' object has no attribute 'Jenkins'
Any ideas?
Edit 2 :
I tries solely python-jenkins API and tried their own example as you see here http://python-jenkins.readthedocs.org/en/latest/example.html
import jenkins
j = jenkins.Jenkins('http://your_url_here', 'username', 'password')
j.get_jobs()
j.create_job('empty', jenkins.EMPTY_CONFIG_XML)
j.disable_job('empty')
j.copy_job('empty', 'empty_copy')
j.enable_job('empty_copy')
j.reconfig_job('empty_copy', jenkins.RECONFIG_XML)
Even this fails at jenkins.Jenkins with attribute error at Jenkins - No module.
I am pretty sure the API is broken.
Your script is probably importing wrong module. You can check it as follows:
import jenkins
print jenkins.__file__
If printed path is other than installation path of jenkins module (eg. C:\Python27_32\lib\site-packages\jenkins\__init__.pyc), then you should check pythonpath:
import sys
print sys.path
Common problem is existence of python script with same name as imported module in current directory, which is at the first place in search path ''.
For more info on import order see module search path
Following #Chemik answer, I realized that the script I wrote was named jenkins.py and it was conflicting with python-jenkins import.
The library isn't broken. Check your script name.
had to add another solution, while running the same command
server = jenkins.Jenkins(jenkins_url, username = '', password = '')
i got the error:
'jenkins' has no attribute 'Jenkins'
my mistake was when installing the package, i installed package "jenkins" and the package i was needed is "python-jenkins".
docs can be found:
python-jenkins docs
so what i had to do is just
pip install python-jenkins