Related
I've a monorepo that contains a set of Python AWS lambdas and I'm using Bazel for building and packaging the lambdas. I'm now trying to use Bazel to create a zip file that follows the expected AWS Lambdas packaging and that I can upload to Lambda. Wondering what's the best way to do this with Bazel?
Below are a few different things I've tried thus far:
Attempt 1: py_binary
BUILD.bazel
py_binary(
name = "main_binary",
srcs = glob(["*.py"]),
main = "main.py",
visibility = ["//appcode/api/transaction_details:__subpackages__"],
deps = [
requirement("Faker"),
],
)
Problem:
This generates the following:
main_binary (python executable)
main_binary.runfiles
main_binary.runfiles_manifest
Lambda expects the handler to be in the format of lambda_function.lambda_handler. Since main_binary is an executable vs. a python file, it doesn't expose the actual handler method and the lambda blows up because it can't find it. I tried updating the handler configuration to simply point to the main_binary but it blows up because it expects two arguments(i.e. lambda_function.lambda_handler).
Attempt 2: py_library + pkg_zip
BUILD.bazel
py_library(
name = "main",
srcs = glob(["*.py"]),
visibility = ["//appcode/api/transaction_details:__subpackages__"],
deps = [
requirement("Faker"),
],
)
pkg_zip(
name = "main_zip",
srcs =["//appcode/api/transaction_details/src:main" ],
)
Problem:
This generates a zip file with:
main.py
__init__.py
The zip file now includes the main.py but none of its runtime dependencies. Thus the lambda blows up because it can't find Faker.
Other Attempts:
I've also tried using the --build_python_zip flag as well as the #bazel_tools//tools/zip:zipper with a generic rule but they both lead to similar outcomes as the two previous attempts.
We use #bazel_tools//tools/zip:zipper with a custom rule. We also pull serverless in using rules_nodejs and run it through bazel, which causes the package building to happen prior to running sls deploy.
We use pip_parse from rules_python. I'm not sure whether the _short_path function below will work with pip_install or other mechanisms.
File filtering is supported, although it's awkward. Ideally the zip generation would be handled by a separate binary (i.e., a Python script) which would allow filtering using regular expressions/globs/etc. Bazel doesn't support regular expressions in Starlark, so we use our own thing.
I've included an excerpt:
lambda.bzl
"""
Support for serverless deployments.
"""
def contains(pattern):
return "contains:" + pattern
def startswith(pattern):
return "startswith:" + pattern
def endswith(pattern):
return "endswith:" + pattern
def _is_ignored(path, patterns):
for p in patterns:
if p.startswith("contains:"):
if p[len("contains:"):] in path:
return True
elif p.startswith("startswith:"):
if path.startswith(p[len("startswith:"):]):
return True
elif p.startswith("endswith:"):
if path.endswith(p[len("endswith:"):]):
return True
else:
fail("Invalid pattern: " + p)
return False
def _short_path(file_):
# Remove prefixes for external and generated files.
# E.g.,
# ../py_deps_pypi__pydantic/pydantic/__init__.py -> pydantic/__init__.py
short_path = file_.short_path
if short_path.startswith("../"):
second_slash = short_path.index("/", 3)
short_path = short_path[second_slash + 1:]
return short_path
def _py_lambda_zip_impl(ctx):
deps = ctx.attr.target[DefaultInfo].default_runfiles.files
f = ctx.outputs.output
args = []
for dep in deps.to_list():
short_path = _short_path(dep)
# Skip ignored patterns
if _is_ignored(short_path, ctx.attr.ignore):
continue
args.append(short_path + "=" + dep.path)
ctx.actions.run(
outputs = [f],
inputs = deps,
executable = ctx.executable._zipper,
arguments = ["cC", f.path] + args,
progress_message = "Creating archive...",
mnemonic = "archiver",
)
out = depset(direct = [f])
return [
DefaultInfo(
files = out,
),
OutputGroupInfo(
all_files = out,
),
]
_py_lambda_zip = rule(
implementation = _py_lambda_zip_impl,
attrs = {
"target": attr.label(),
"ignore": attr.string_list(),
"_zipper": attr.label(
default = Label("#bazel_tools//tools/zip:zipper"),
cfg = "host",
executable = True,
),
"output": attr.output(),
},
executable = False,
test = False,
)
def py_lambda_zip(name, target, ignore, **kwargs):
_py_lambda_zip(
name = name,
target = target,
ignore = ignore,
output = name + ".zip",
**kwargs
)
BUILD.bazel
load("#npm_serverless//serverless:index.bzl", "serverless")
load(":lambda.bzl", "contains", "endswith", "py_lambda_zip", "startswith")
py_binary(
name = "my_lambda_app",
...
)
py_lambda_zip(
name = "lambda_archive",
ignore = [
contains("/__pycache__/"),
endswith(".pyc"),
endswith(".pyo"),
# Ignore boto since it's provided by Lambda.
startswith("boto3/"),
startswith("botocore/"),
# With the move to hermetic toolchains, the zip gets a lib/ directory containing the
# python runtime. We don't need that.
startswith("lib/"),
],
target = ":my_lambda_app",
# Only allow building on linux, since we don't want to upload a lambda zip file
# with e.g. macos compiled binaries.
target_compatible_with = [
"#platforms//os:linux",
],
)
# The sls command requires that serverless.yml be in its working directory, and that the yaml file
# NOT be a symlink. So this target builds a directory containing a copy of serverless.yml, and also
# symlinks the generated lambda_archive.zip in the same directory.
#
# It also generates a chdir.js script that we instruct node to execute to change to the proper working directory.
genrule(
name = "sls_files",
srcs = [
"lambda_archive.zip",
"serverless.yml",
],
outs = [
"sls_files/lambda_archive.zip",
"sls_files/serverless.yml",
"sls_files/chdir.js",
],
cmd = """
mkdir -p $(#D)/sls_files
cp $(location serverless.yml) $(#D)/sls_files/serverless.yml
cp -P $(location lambda_archive.zip) $(#D)/sls_files/lambda_archive.zip
echo "const fs = require('fs');" \
"const path = require('path');" \
"process.chdir(path.dirname(fs.realpathSync(__filename)));" > $(#D)/sls_files/chdir.js
""",
)
# Usage:
# bazel run //:sls -- deploy <more args>
serverless(
name = "sls",
args = ["""--node_options=--require=./$(location sls_files/chdir.js)"""],
data = [
"sls_files/chdir.js",
"sls_files/serverless.yml",
"sls_files/lambda_archive.zip",
],
)
serverless.yml
service: my-app
package:
artifact: lambda_archive.zip
# ... other config ...
Below are the changes I made to the previous answer to generate the lambda zip. Thanks #jvolkman for the original suggestion.
project/BUILD.bazel: Added rule to generate requirements_lock.txt from project/requirements.txt
load("#rules_python//python:pip.bzl", "compile_pip_requirements")
compile_pip_requirements(
name = "requirements",
extra_args = ["--allow-unsafe"],
requirements_in = "requirements.txt",
requirements_txt = "requirements_lock.txt",
)
project/WORKSPACE.bazel: swap pip_install with pip_parse
workspace(name = "mdc-eligibility")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_python",
sha256 = "9fcf91dbcc31fde6d1edb15f117246d912c33c36f44cf681976bd886538deba6",
strip_prefix = "rules_python-0.8.0",
url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.8.0.tar.gz",
)
load("#rules_python//python:repositories.bzl", "python_register_toolchains")
python_register_toolchains(
name = "python3_9",
python_version = "3.9",
)
load("#rules_python//python:pip.bzl", "pip_parse")
load("#python3_9//:defs.bzl", "interpreter")
pip_parse(
name = "mndc-eligibility-deps",
requirements_lock = "//:requirements_lock.txt",
python_interpreter_target = interpreter,
quiet = False
)
load("#mndc-eligibility-deps//:requirements.bzl", "install_deps")
install_deps()
project/build_rules/lambda_packaging/lambda.bzl: Modified custom rule provided by #jvolkman to include source code in the resulting zip code.
def contains(pattern):
return "contains:" + pattern
def startswith(pattern):
return "startswith:" + pattern
def endswith(pattern):
return "endswith:" + pattern
def _is_ignored(path, patterns):
for p in patterns:
if p.startswith("contains:"):
if p[len("contains:"):] in path:
return True
elif p.startswith("startswith:"):
if path.startswith(p[len("startswith:"):]):
return True
elif p.startswith("endswith:"):
if path.endswith(p[len("endswith:"):]):
return True
else:
fail("Invalid pattern: " + p)
return False
def _short_path(file_):
# Remove prefixes for external and generated files.
# E.g.,
# ../py_deps_pypi__pydantic/pydantic/__init__.py -> pydantic/__init__.py
short_path = file_.short_path
if short_path.startswith("../"):
second_slash = short_path.index("/", 3)
short_path = short_path[second_slash + 1:]
return short_path
# steven chambers
def _py_lambda_zip_impl(ctx):
deps = ctx.attr.target[DefaultInfo].default_runfiles.files
f = ctx.outputs.output
args = []
for dep in deps.to_list():
short_path = _short_path(dep)
# Skip ignored patterns
if _is_ignored(short_path, ctx.attr.ignore):
continue
args.append(short_path + "=" + dep.path)
# MODIFICATION: Added source files to the map of files to zip
source_files = ctx.attr.target[DefaultInfo].files
for source_file in source_files.to_list():
args.append(source_file.basename+"="+source_file.path)
ctx.actions.run(
outputs = [f],
inputs = deps,
executable = ctx.executable._zipper,
arguments = ["cC", f.path] + args,
progress_message = "Creating archive...",
mnemonic = "archiver",
)
out = depset(direct = [f])
return [
DefaultInfo(
files = out,
),
OutputGroupInfo(
all_files = out,
),
]
_py_lambda_zip = rule(
implementation = _py_lambda_zip_impl,
attrs = {
"target": attr.label(),
"ignore": attr.string_list(),
"_zipper": attr.label(
default = Label("#bazel_tools//tools/zip:zipper"),
cfg = "host",
executable = True,
),
"output": attr.output(),
},
executable = False,
test = False,
)
def py_lambda_zip(name, target, ignore, **kwargs):
_py_lambda_zip(
name = name,
target = target,
ignore = ignore,
output = name + ".zip",
**kwargs
)
project/appcode/api/transaction_details/src/BUILD.bazel: Used custom py_lambda_zip rule to zip up py_library
load("#mndc-eligibility-deps//:requirements.bzl", "requirement")
load("#python3_9//:defs.bzl", "interpreter")
load("//build_rules/lambda_packaging:lambda.bzl", "contains", "endswith", "py_lambda_zip", "startswith")
py_library(
name = "main",
srcs = glob(["*.py"]),
visibility = ["//appcode/api/transaction_details:__subpackages__"],
deps = [
requirement("Faker"),
],
)
py_lambda_zip(
name = "lambda_archive",
ignore = [
contains("/__pycache__/"),
endswith(".pyc"),
endswith(".pyo"),
# Ignore boto since it's provided by Lambda.
startswith("boto3/"),
startswith("botocore/"),
# With the move to hermetic toolchains, the zip gets a lib/ directory containing the
# python runtime. We don't need that.
startswith("lib/"),
],
target = ":main",
)
I create 30 instance to run crawl script, and want to stop aws instances after complete running this command, python3 aws_crawl.py, but I cannot find how to stop current aws instance by command line.
Please see my main.tf code below:
resource "aws_instance" "crawl_worker" {
ami = "${var.ami_id}"
instance_type = "t3a.nano"
security_groups = ["${var.aws_security_group}"]
key_name = "secret"
count = 10
tags = {
Name = "crawl_worker_${count.index}"
}
connection {
type = "ssh"
host = "${self.public_ip}"
user = "ubuntu"
private_key = "${file("~/secret.pem")}"
timeout = "50m"
}
provisioner "remote-exec" {
inline = [
"python3 aws_crawl.py"
]
}
}
I am trying to convert this code:
import pandas as pd
import matplotlib.pyplot as plt
import readTrc
path = 'C:/filepath/data.trc'
datX, datY, m = readTrc.readTrc(path)
srx, sry = pd.Series(datX * 1000), pd.Series(datY * 1000)
df = pd.concat([srx, sry], axis = 1)
df.set_index(0, inplace = True)
df.plot(grid = 1,
linewidth = 0.5,
figsize = (9,5),
legend = False,
xlim = (df.index[0] , df.index[-1]),
xticks = [-9,-8,-7,-6,-5,-4,-3,-2,-1,0,1,2,3,4,5,6,7,8,9])
plt.xlabel('Zeit in ms')
plt.ylabel('Spannung in mV')
plt.savefig('test.png', dpi = 600)
into an executable with cx_Freeze.
Setup.py :
import cx_Freeze
import sys
import matplotlib
base = None
if sys.platform == "win32":
base = "Win32GUI"
executables = [
cx_Freeze.Executable("read_trc.py", base = base),
]
build_exe_options = {"includes":["matplotlib.backends.backend_tkagg"],
"include_files":[(matplotlib.get_data_path(), "mpl-data"),
('C:/filepath', 'data.trc')],
"excludes":[],
}
cx_Freeze.setup(
name = "script",
options = {"build_exe": build_exe_options},
version = "0.0",
description = "A basic example",
executables = executables)
The conversion works but when I try to run the .exe I get this error:
Is there a way to make this work? I am using Python 3.6 on Windows 10.
I have already tried the fixes found on Stackoverflow regarding Numpy import errors but it does not seem to help.
Edit:
Thanks to comments I solved the Error. Unfortunately now I get another Error when I try to execute the converted file:
Edit2:
I tried to include tkinter in my code but it doesn't work. I have a standard python 3.6 distribution installed which should include tkinter. tkinter.test() works. I assume that there is something wrong with the tkinter.ddls import. How would I do that correctly?
import cx_Freeze
import sys
import os
import matplotlib
os.environ['TCL_LIBRARY'] = r"C:\Users\Artur\AppData\Local\Programs\Python\Python36\DLLs\tcl86t.dll"
os.environ['TK_LIBRARY'] = r"C:\Users\Artur\AppData\Local\Programs\Python\Python36\DLLs\tk86t.dll"
base = None
if sys.platform == "win32":
base = "Win32GUI"
executables = [
cx_Freeze.Executable("plt_test.py", base = base),
]
include_files = [(matplotlib.get_data_path(), "mpl-data"),
('C:/Users/Artur/Desktop/Nellingen/Messdaten/20180104_Zyl_NiCr_DCneg_5bar_HSC/20180104_04_Zyl_NiCr_DCneg_5bar_170kV_HSC_Firefly/C220180104_ch2_UHF00000.trc',
'C220180104_ch2_UHF00000.trc'),
(r"C:\Users\Artur\AppData\Local\Programs\Python\Python36\DLLs\tcl86t.dll",
r"C:\Users\Artur\AppData\Local\Programs\Python\Python36\DLLs\tk86t.dll")]
build_exe_options = {"includes":["matplotlib.backends.backend_tkagg"],
"include_files":include_files,
"excludes":[],
"packages":["numpy", "matplotlib", "pandas", 'tkinter', 'os'],
}
cx_Freeze.setup(
name = "script",
options = {"build_exe": build_exe_options},
version = "0.0",
description = "A basic example",
executables = executables)
Output in windows console:
I'm new to bazel and trying to automatize updates in the MapIt database and code.
I named the root folder for my bazel project bazel_mapit where there is the WORKSPACE file and the BUILD file for this external dependency as follows.
WORSPACE:
new_git_repository(
name = "mapit_repo",
remote = "https://github.com/mysociety/mapit",
tag = "v2.0",
build_file = "mapit_repo.BUILD",
init_submodules = 1,
)
mapit_reop.BUILD:
package(default_visibility = ["//visibility:public"])
filegroup(
name = "mapit_files",
srcs = glob(
[
"**/*",
],
exclude = [
"**/LICENSE",
"**/*.zip",
],
),
)
Now I want to generate the configuration file conf/general.yml inside the MapIt sources. The problem now is, when I add the following code to the mapit_repo.BUILD file, I getting errors that bazel can't find
'#bazel_mapit//:general_yml.bzl'
mapit_repo.BUILD (extension):
load("#bazel_mapit//:general_yml.bzl", "general_yml")
general_yml(
name = 'generate_general_yml',
bzl_mapit_db_user = 'foo',
)
filegroup (
name = 'mapit_general_yml',
srcs = ['conf/general.yml'],
data = ['conf/general.yml-example'],
)
How can I generate a config file in an external dependency?
Update:
This is the content of the working mapit_repo.BUILD file:
package(default_visibility = ["//visibility:public"])
filegroup(
name = "mapit_files",
srcs = glob(
[
"**/*",
],
exclude = [
"**/LICENSE",
"**/*.zip",
],
),
)
load("#//conf:general_yml.bzl", "general_yml")
general_yml(
name = 'generate_general_yml',
bzl_mapit_db_user = 'foo',
)
filegroup (
name = 'mapit_general_yml',
srcs = ['conf/general.yml'],
data = ['#//conf:general.yml.tmpl'],
)
I think all that needs to change is the load() mapit_repo.BUILD. That build file is going to be evaluated from within the external workspace, so to reference the bzl file in the main workspace, you can use #//:general_yml.bzl. (That is, the empty workspace name refers to the main workspace)
I am getting an unexpected error. I realize that there are posts with similar errors but either could not understand the answer or could not relate it to my case (dictionary).
I am trying to calculate a similarity score for each line of an input file and at every iteration (i.e for each line of input file) store the top 20 values of the score in a dictionary.
Following is my code:
import sys
from cx_Freeze import setup, Executable
includefiles = ['Arcade Funk.mp3', 'game over.wav', 'FrogTown.wav','pixel ufo.png','introBackground.png','pixel playButton.png','pixel instructionButton.png','pixel playButtonHighlighted.png','pixel instructionButtonHighlighted.png','instructionPage.png','crashBackground.png','space background long.png','pixel earth.png','pixel asteroid.png', 'pixel icon.png','Montserrat-ExtraBold.otf','Montserrat-Bold.otf','arial.ttf']
includes = []
excludes = ['Tkinter']
packages = ['pygame']
build_exe_options = {'includes':[includes],'packages':[packages], 'excludes':[excludes], 'include_files':[includefiles]}
base = None
if sys.platform == 'win64':
base = 'Win64GUI'
elif sys.platform == 'win32':
base = 'Win32GUI'
setup( name = 'Earth Invaders',
version = '0.1',
author = 'redacted',
description = 'Slider Game: Space',
options = {'build_exe': [build_exe_options]},
executables = [Executable('EarthInvaders.py', base=base)]
)
This is the error
Traceback (most recent call last):
File "C:/Users/Vix_Ox/Desktop/Earth Invaders/setup.py", line 21, in <module>
executables = [Executable('EarthInvaders.py', base=base)]
File "C:\Users\----\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\cx_Freeze\dist.py", line 349, in setup
distutils.core.setup(**attrs)
File "C:\Users\----\AppData\Local\Programs\Python\Python36-32\lib\distutils\core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "C:\Users\----\AppData\Local\Programs\Python\Python36-32\lib\site-packages\cx_Freeze\dist.py", line 24, in __init__
distutils.dist.Distribution.__init__(self, attrs)
File "C:\Users\----\AppData\Local\Programs\Python\Python36-32\lib\distutils\dist.py", line 237, in __init__
for (opt, val) in cmd_options.items():
AttributeError: 'list' object has no attribute 'items'
It looks like you've been following the documentation fine.
I think the issue is some extra square braces on line 20: [build_exe_options] should be build_exe_options. That variable is expected to be a dictionary, but it's getting a list, thus the error.
setup( name = 'Earth Invaders',
version = '0.1',
author = 'redacted',
description = 'Slider Game: Space',
options = {'build_exe': build_exe_options},
executables = [Executable('EarthInvaders.py', base=base)]
)
You may also find that you have to apply this retroactively to an earlier line, as they are already encapsulated in lists when they are declared:
build_exe_options = {'includes':includes,'packages':packages, 'excludes':excludes, 'include_files':includefiles}