We Keep Coding

Unable to implement ICodePipelineActionFactory in Python

I am trying to create an arbitrary CodePipeline action as part of a CDK pipeline implemented in Python. Specifically in this case it's a step function invocation, but I would like to call other types as well. No matter what I do, I keep getting the same error saying it can't find the add_action attribute on the stage object.
jsii.errors.JSIIError: '' object has no attribute 'add_action'
I have tried different variations of the method name, checking the object with dir() (stage is a very opaque InterfaceDynamicProxy object), reading jsii documentation to see if they have a way to list available attributes, but got nowhere.
Does anyone have a working example of jsii interface implementation in Python? Or can you tell what's wrong with the code below?
I am using CDK 1.118.0 with Python 3.9.6 and node.js 16.6.2 on Mac OS X.
from aws_cdk import core, pipelines, aws_codepipeline_actions, aws_codepipeline, aws_stepfunctions
import jsii
#jsii.implements(pipelines.ICodePipelineActionFactory)
class SomeStep(pipelines.Step):
def __init__(self, id_):
super().__init__(id_)
#jsii.member(jsii_name="produceAction")
def produce_action(
self, stage: aws_codepipeline.IStage,
options: pipelines.ProduceActionOptions,
# TODO why are these not passed?
# *,
# action_name, artifacts, pipeline, run_order, scope,
# before_self_mutation=None,
# code_build_defaults=None,
# fallback_artifact=None
) -> pipelines.CodePipelineActionFactoryResult:
stage.add_action(
aws_codepipeline_actions.StepFunctionInvokeAction(
state_machine=aws_stepfunctions.StateMachine.from_state_machine_arn("..."),
action_name="foo",
state_machine_input=aws_codepipeline_actions.StateMachineInput.literal({"foo": "bar"}),
run_order=options["run_order"],
)
)
return pipelines.CodePipelineActionFactoryResult(run_orders_consumed=1)
app = core.App()
stage = core.Stage(app, "stage")
stack = core.Stack(stage, "stack")
pipeline_stack = core.Stack(app, "pipeline-stack")
pipeline = pipelines.CodePipeline(
pipeline_stack,
"pipeline",
synth=pipelines.ShellStep("synth", input=pipelines.CodePipelineSource.git_hub("foo/bar", "main"), commands=["cdk synth"])
)
pipeline.add_wave("wave").add_stage(stage, pre=[SomeStep("some")])
app.synth()
The complete error:
jsii.errors.JavaScriptError:
Error: '' object has no attribute 'add_action'
at KernelHost.completeCallback (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/tmpwwmvzicu/lib/program.js:9462:35)
at KernelHost.callbackHandler (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/tmpwwmvzicu/lib/program.js:9453:41)
at Step.value (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/tmpwwmvzicu/lib/program.js:8323:49)
at CodePipeline.pipelineStagesAndActionsFromGraph (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/jsii-kernel-x3iY7A/node_modules/#aws-cdk/pipelines/lib/codepipeline/codepipeline.js:154:48)
at CodePipeline.doBuildPipeline (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/jsii-kernel-x3iY7A/node_modules/#aws-cdk/pipelines/lib/codepipeline/codepipeline.js:116:14)
at CodePipeline.buildPipeline (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/jsii-kernel-x3iY7A/node_modules/#aws-cdk/pipelines/lib/main/pipeline-base.js:93:14)
at CodePipeline.buildJustInTime (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/jsii-kernel-x3iY7A/node_modules/#aws-cdk/pipelines/lib/main/pipeline-base.js:101:18)
at Object.visit (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/jsii-kernel-x3iY7A/node_modules/#aws-cdk/pipelines/lib/main/pipeline-base.js:42:57)
at recurse (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/jsii-kernel-x3iY7A/node_modules/#aws-cdk/core/lib/private/synthesis.js:86:20)
at recurse (/private/var/folders/ln/r1dlp_xj6t57ddclvh7zgl8m0000gp/T/jsii-kernel-x3iY7A/node_modules/#aws-cdk/core/lib/private/synthesis.js:98:17)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "~/dev/cdk-playground/app.py", line 55, in <module>
app.synth()
File ".../lib/python3.9/site-packages/aws_cdk/core/__init__.py", line 16432, in synth
return typing.cast(aws_cdk.cx_api.CloudAssembly, jsii.invoke(self, "synth", [options]))
File ".../lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 128, in wrapped
return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
File ".../lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 348, in invoke
return _callback_till_result(self, response, InvokeResponse)
File ".../lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 216, in _callback_till_result
response = kernel.sync_complete(
File ".../lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 386, in sync_complete
return self.provider.sync_complete(
File ".../lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 382, in sync_complete
resp = self._process.send(_CompleteRequest(complete=request), response_type)
File ".../lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 326, in send
raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: '' object has no attribute 'add_action'
Subprocess exited with error 1

AWS has resolved the issue in jsii.
https://github.com/aws/jsii/issues/2963
It should be available with CDK 1.121.0.

Getting KeyError: 'Endpoint' error in Python when calling Custom Vision API

from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
from msrest.authentication import CognitiveServicesCredentials
from azure.cognitiveservices.vision.customvision import prediction
from PIL import Image
endpoint = "https://southcentralus.api.cognitive.microsoft.com/"
project_id = "projectidhere"
prediction_key = "predictionkeyhere"
predict = CustomVisionPredictionClient(prediction_key, endpoint)
with open("c:/users/paul.barbin/pycharmprojects/hw3/TallowTest1.jpg", mode="rb") as image_data:
tallowresult = predict.detect_image(project_id, "test1", image_data)
Python 3.7, and I'm using Azure Custom Vision 3.1? (>azure.cognitiveservices.vision.customvision) (3.1.0)
Note that I've seen the same question on SO but no real solution. The posted answer on the other question says to use the REST API instead.
I believe the error is in the endpoint (as stated in the error), and I've tried a few variants - with the slash, without, using an environment variable, without, I've tried appending various strings to my endpoint but I keep getting the same message. Any help is appreciated.
Full error here:
Traceback (most recent call last):
File "GetError.py", line 15, in <module>
tallowresult = predict.detect_image(project_id, "test1", image_data)
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\azure\cognitiveservices\vision\customvision\prediction\operations\_custom_vision_
prediction_client_operations.py", line 354, in detect_image
request = self._client.post(url, query_parameters, header_parameters, form_content=form_data_content)
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\msrest\service_client.py", line 193, in post
request = self._request('POST', url, params, headers, content, form_content)
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\msrest\service_client.py", line 108, in _request
request = ClientRequest(method, self.format_url(url))
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\msrest\service_client.py", line 155, in format_url
base = self.config.base_url.format(**kwargs).rstrip('/')
KeyError: 'Endpoint'

CustomVisionPredictionClient takes two required, positional parameters: endpoint and credentials. Endpoint needs to be passed in before credentials, try swapping the order:
predict = CustomVisionPredictionClient(endpoint, prediction_key)

Using Python to Manage AWS

I’m trying to use Python to create EC2 instances but I keep getting these errors.
Here is my code:
#!/usr/bin/env python
import boto3
ec2 = boto3.resource('ec2')
instance = ec2.create_instances(
ImageId='ami-0922553b7b0369273',
MinCount=1,
MaxCount=1,
InstanceType='t2.micro')
print instance[0].id
Here are the errors I'm getting
Traceback (most recent call last):
File "./createinstance.py", line 8, in <module>
InstanceType='t2.micro')
File "/usr/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation: The image id '[ami-0922553b7b0369273]' does not exist
I also get an error when trying to create a key pair
Here's my code for creating the keypair
import boto3
ec2 = boto3.resource('ec2')
# create a file to store the key locally
outfile = open('ec2-keypair.pem','w')
# call the boto ec2 function to create a key pair
key_pair = ec2.create_key_pair(KeyName='ec2-keypair')
# capture the key and store it in a file
KeyPairOut = str(key_pair.key_material)
print(KeyPairOut)
outfile.write(KeyPairOut)
response = ec2.instance-describe()
print response
Here's are the error messages
./createkey.py: line 1: import: command not found
./createkey.py: line 2: syntax error near unexpected token `('
./createkey.py: line 2: `ec2 = boto3.resource('ec2')'
What I am I missing?

For your first script, one of two possibilities could be occurring:
1. The AMI you are referencing by the ID is not available because the key is incorrect or the AMI doesn't exist
2. AMI is unavailable in the region that your machine is setup for
You most likely are running your script from a machine that is not configured for the correct region. If you are running your script locally or on a server that does not have roles configured, and you are using the aws-cli, you can run the aws configure command to configure your access keys and region appropriately. If you are running your instance on a server with roles configured, your server needs to be ran in the correct region, and your roles need to allow access to EC2 AMI's.
For your second question (which in the future should probably be posted separate), your syntax error in your script is a side effect of not following the same format for how you wrote your first script. It is most likely that your python script is not in fact being interpreted as a python script. You should add the shebang at the top of the file and remove the spacing preceding your import boto3 statement.
#!/usr/bin/env python
import boto3
# create a file to store the key locally
outfile = open('ec2-keypair.pem','w')
# call the boto ec2 function to create a key pair
key_pair = ec2.create_key_pair(KeyName='ec2-keypair')
# capture the key and store it in a file
KeyPairOut = str(key_pair.key_material)
print(KeyPairOut)
outfile.write(KeyPairOut)
response = ec2.instance-describe()
print response

Unit testing bottle py application that uses request body results in KeyError: 'wsgi.input'

When unit testing a bottle py route function:
from bottle import request, run, post
#post("/blah/<boo>")
def blah(boo):
body = request.body.readline()
return "body is %s" % body
blah("booooo!")
The following exception is raised:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in blah
File "bottle.py", line 1197, in body
self._body.seek(0)
File "bottle.py", line 166, in __get__
if key not in storage: storage[key] = self.getter(obj)
File "bottle.py", line 1164, in _body
read_func = self.environ['wsgi.input'].read
KeyError: 'wsgi.input'
The code will work if running as a server via bottle's run function, it's purely when I call it as a normal Python function e.g. in a unit test.
What am I missing? How can I invoke this as a normal python func inside a unit test?

I eventually worked out what the problem is. I needed to "fake" the request environment for bottle to play nicely:
from bottle import request, run, post, tob
from io import BytesIO
body = "abc"
request.environ['CONTENT_LENGTH'] = str(len(tob(body)))
request.environ['wsgi.input'] = BytesIO()
request.environ['wsgi.input'].write(tob(body))
request.environ['wsgi.input'].seek(0)
# Now call your route function and assert

Another issue is that Bottle uses thread locals and reads the BytesIO object you put into request.environ the first time you access the body property on request. Therefore if you run multiple tests with post data, e.g. in a TestCase, when you come to read it in your request callback it will only return the value it was initially given, not your updated value.
The solution is to scrub all the values stored on the request object before each test, so in your setUp(self) you can do something like this:
class MyTestCase(TestCase):
def setUp():
# Flush any cached values
request.bind({})

Check out https://pypi.python.org/pypi/boddle. In your test you could do:
from bottle import request, run, post
from boddle import boddle
#post("/blah/<boo>")
def blah(boo):
body = request.body.readline()
return "body is %s" % body
with boddle(body='woot'):
print blah("booooo!")
Which would print body is woot.
Disclaimer: I authored. (Wrote it for work.)

How to fix IncompleteRead error on Linux using Py2Neo

I am updating data on a Neo4j server using Python (2.7.6) and Py2Neo (1.6.4). My load function is:
from py2neo import neo4j,node, rel, cypher
session = cypher.Session('http://my_neo4j_server.com.mine:7474')
def load_data():
tx = session.create_transaction()
for row in dataframe.iterrows(): #dataframe is a pandas dataframe
name = row[1].name
id = row[1].id
merge_query = "MERGE (a:label {name:'%s', name_var:'%s'}) " % (id, name)
tx.append(merge_query)
tx.commit()
When I execute this from Spyder in Windows it works great. All the data from the dataframe is committed to neo4j and visible in the graph. However, when I run this from a linux server (different from the neo4j server) I get the following error at tx.commit(). Note that I have the same version of python and py2neo.
INFO:py2neo.packages.httpstream.http:>>> POST http://neo4j1.qs:7474/db/data/transaction/commit [1360120]
INFO:py2neo.packages.httpstream.http:<<< 200 OK [chunked]
ERROR:__main__:some part of process failed
Traceback (most recent call last):
File "my_file.py", line 132, in load_data
tx.commit()
File "/usr/local/lib/python2.7/site-packages/py2neo/cypher.py", line 242, in commit
return self._post(self._commit or self._begin_commit)
File "/usr/local/lib/python2.7/site-packages/py2neo/cypher.py", line 208, in _post
j = rs.json
File "/usr/local/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 563, in json
return json.loads(self.read().decode(self.encoding))
File "/usr/local/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 634, in read
data = self._response.read()
File "/usr/local/lib/python2.7/httplib.py", line 543, in read
return self._read_chunked(amt)
File "/usr/local/lib/python2.7/httplib.py", line 597, in _read_chunked
raise IncompleteRead(''.join(value))
IncompleteRead: IncompleteRead(128135 bytes read)
This post (IncompleteRead using httplib) suggests that is an httplib error. I am not sure how to handle since I am not calling httplib directly.
Any suggestions for getting this load to work on Linux or what the IncompleteRead error message means?
UPDATE :
The IncompleteRead error is being caused by a Neo4j error being returned. The line returned in _read_chunked that is causing the error is:
pe}"}]}],"errors":[{"code":"Neo.TransientError.Network.UnknownFailure"
Neo4j docs say this is an unknown network error.

Although I can't say for sure, this implies some kind of local network issue between client and server rather than a bug within the library. Py2neo wraps httplib (which is pretty solid itself) and, from the stack trace, it looks as though the client is expecting more chunks from a chunked response.
To diagnose further, you could make some curl calls from your Linux application server to your database server and see what succeeds and what doesn't. If that works, try writing a quick and dirty python script to make the same calls with httplib directly.
UPDATE 1: Given the update above and the fact that the server streams its responses, I'm thinking that the chunk size might represent the intended payload but the error cuts the response short. Recreating the issue with curl certainly seems like the best next step to help determine whether it is a fault in the driver, the server or something else.
UPDATE 2: Looking again this morning, I notice that you're using Python substitution for the properties within the MERGE statement. As good practice, you should use parameter substitution at the Cypher level:
merge_query = "MERGE (a:label {name:{name}, name_var:{name_var}})"
merge_params = {"name": id, "name_var": name}
tx.append(merge_query, merge_params)