We Keep Coding

Create instanceprofile with CDK and Python

I am new to cdk and trying to create an instance profile with CDK+Python with the following code. I have already created the Role (gitLabRunner-glue) successfully thru CDK and wanting to use it with the intance profile. However, when i run the following code, i get an error gitLabRunner-glue already exists
Can somebody please explain what am i missing ?
from aws_cdk import core as cdk
from aws_cdk import aws_glue as glue
from aws_cdk import aws_ec2 as _ec2
from aws_cdk import aws_iam as _iam
class Ec2InstanceProfile(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
# role = _iam.Role(self, "instanceprofilerole", role_name="gitLabRunner-glue",
# assumed_by=_iam.ServicePrincipal(service='ec2.amazonaws.com'))
ec2gitLabRunnerinstanceprofile = _iam.CfnInstanceProfile(
self,
"ec2gitLabRunnerinstanceprofile",
instance_profile_name="ec2-gitLabRunner-glue",
roles=["gitLabRunner-glue"] # also tried with this[role.role_name]
)

Does your AWS account already have a role with that name in it?
the Cfn Functions in cdk represent constructs and services that have not been fully hooked into all that is CDK. As such, they often don't do things that others would - where as a CloudFormation Template for the instance profile may just hook into the existing role, the coding in the back of this cfn function may go ahead and create a role item in the template output.
if you do a cdk synth, look in your cdk.out directory, find your cloudformation template, then do a search for gitLabRunner-glue - you may find there is a AWS::IAM::ROLE being created, indicating when CloudFormation attempts to run based of the template created by cdk it tries to create a new resource and it cant.
You have a couple options to try:
As you tried, uncomment the role again and use role.role_name but name the role something else or, as CDK recommends, don't include a name and let it name it for you
search your aws account for the role and delete it
If you absolutely cannot delete the existing role or cannot create a new one with a new name, then import the role, using (based off your imports)
role = _iam.Role.from_role_arn(self, "ImportedGlueRole", role_arn="arn:aws:of:the:role", add_grants_to_resources=True)
be wary a bit of the add_grants_to_resources - if its not your role to mess with cdk can make changes if you make that true and that could cause issues elsewhere - but if its not true, then you have to update the Role itself in the aws console (or cli) to accept your resources as able to assume it.

I made it work like this, not the desired model though, but given the limitations of cdk, i couldn't find any other way.
from aws_cdk import core as cdk
from aws_cdk import aws_glue as glue
from aws_cdk import aws_ec2 as _ec2
from aws_cdk import aws_iam as _iam
class Ec2InstanceProfile(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
boundary = _iam.ManagedPolicy.from_managed_policy_arn(self, "Boundary",
"arn:aws:iam::${ACCOUNT_ID}:policy/app-perm-boundary")
# role = _iam.Role(self, "instanceprofilerole", role_name="gitLabRunner-glue",
# assumed_by=_iam.ServicePrincipal(service='ec2.amazonaws.com'))
ec2_gitlabrunner_glue = _iam.Role(
self, 'ec2-gitlabrunner-glue',
role_name='gitLabRunner-glue',
description='glue service role to be attached to the runner',
# inline_policies=[write_to_s3_policy],
assumed_by=_iam.ServicePrincipal('ec2.amazonaws.com'),
permissions_boundary=boundary
)
ec2gitLabRunnerinstanceprofile = _iam.CfnInstanceProfile(
self,
"gitLabRunnerinstanceprofile",
instance_profile_name="gitLabRunner-glue",
roles=["gitLabRunner-glue"]
)

How to link typing-like nested classes and other urls with Sphinx and RST

Using intersphinx and autodoc, having:
:param stores: Array of objects
:type stores: list[dict[str,int]]
Would result in an entry like:
stores (list[dict[str,int]]) - Array of objects.
Is there a way to convert list[dict[str,int]] outside of the autodoc :param: derivative (or others like :rtype:) with raw RST (within the docstring) or programatically given a 'list[dict[str,int]]' string?
Additionally, is it possible to use external links within the aforementioned example?
Example
Consider a script.py file:
def some_func(arg1):
"""
This is a head description.
:param arg1: The type of this param is hyperlinked.
:type arg1: list[dict[str,int]]
Is it possible to hyperlink this, here: dict[str,list[int]]
Or even add custom references amongst the classes: dict[int,ref]
Where *ref* links to a foreign, external source.
"""
Now in the Sphinx conf.py file add:
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx'
]
intersphinx_mapping = {
'py': ('https://docs.python.org/3', None),
}
In your index.rst, add:
Title
=====
.. toctree::
:maxdepth: 2
.. autofunction:: script.some_func
And now simply make the html for the page.
The list[dict[str,int]] next to :type arg1: will be hyperlinked as shown at the beginning of this question, but dict[str,list[int]] obviously won't. Is there a way to make the latter behave like the former?

I reached a solution after digging around sphinx's code.
Injecting External References (:param:)
I created a custom extension that connects to the missing-reference event and attempts to resolve unknown references.
Code of reflinks.py:
import docutils.nodes as nodes
_cache = {}
def fill_cache(app):
_cache.update(app.config.reflinks)
def missing_reference(app, env, node, contnode):
target = node['reftarget']
try:
uri = _cache[target]
except KeyError:
return
newnode = nodes.reference('', '', internal = False, refuri = uri)
if not node.get('refexplicit'):
name = target.replace('_', ' ') # style
contnode = contnode.__class__(name, name)
newnode.append(contnode)
return newnode
def setup(app):
app.add_config_value('reflinks', None, False)
app.connect('builder-inited', fill_cache)
app.connect('missing-reference', missing_reference, priority = 1000)
Explanation
I consulted intersphinx's methodology for resolving unknown references and connected the function with high priority so it's hopefully only consulted as a last result.
Followup
Include the extenion.
Adding to conf.py:
reflinks = {'google': 'https://google.com'}
Allowed for script.py:
def some_func(arg1):
"""
:param arg1: Google homepages.
:type arg1: dict[str, google]
"""
Where dict[str, google] are now all hyperlinks.
Formatting Nested Types
There were instances where I wanted to use type structures like list[dict[str,myref]] outside of fields like :param:, :rtype:, etc. Another short extension did the trick.
Code of nestlinks.py:
import sphinx.domains.python as domain
import docutils.parsers.rst.roles as roles
_field = domain.PyTypedField('class')
def handle(name, rawtext, text, lineno, inliner, options = {}, content = []):
refs = _field.make_xrefs('class', 'py', text)
return (refs, [])
def setup(app):
roles.register_local_role('nref', handle)
Explanation
After reading this guide on roles, and digging here and here I realised that all I needed was a dummy field to handle the whole reference-making work and pretend like it's trying to reference classes.
Followup
Include the extension.
Now script.py:
def some_func(arg1):
"""
:param arg1: Google homepages.
:type arg1: dict[str, google]
Now this :nref:`list[dict[str,google]]` is hyperlinked!
"""
Notes
I am using intersphinx and autodoc to link to python's types and document my function's docstrings.
I am not well-versed in Sphinx's underlying mechanisms so take my methodology with a grain of salt.
The examples are provided are adjusted for the sake of being re-usable and generic and have not been tested.
The usability of such features is obviously questionable and only necessary when libraries like extlinks don't cover your needs.

Adding an HTML class to a Sphinx directive with an extension

I'm trying to write a basic extension to create a new directive (output) based on the only directive provided by Sphinx. This new directive simply needs to add an HTML class to the standard result of the Only directive.
So for this I have tried the following code based on this collapse-code-block extension.
Since docutils has virtually no documentation and that I'm not a very experienced python developer, I'm struggling to make this work.
Here is what I have tried, amongst other variations that all led to no real indication on the issue:
from docutils import nodes
from sphinx.directives.other import Only
class output_node(nodes.General, nodes.Element):
pass
class output_directive(Only):
option_spec = Only.option_spec
def run(self):
env = self.state.document.settings.env
node = output_node()
output = Only.run(self)
node.setup_child(output)
node.append(output)
return [node]
def html_visit_output_node(self, node):
self.body.append(self.starttag(node, 'div', '', CLASS='internalonly'))
def html_depart_output_node(self, node):
self.body.append('</div>')
def setup(app):
app.add_node(
output_node,
html=(
html_visit_output_node,
html_depart_output_node
)
)
app.add_directive('output', output_directive)
I don't think it should be more complicated than that but this just doesn't cut it.
Any idea?

How to add Members folder in plone.app.testing?

How can I add Members folder for my functional tests in plone.app.testing so that it is findable as in real site?
Have have set member area creation flag in my product installation step which I'm testing.
membership.memberareaCreationFlag = 1
I need to get this test working:
class TestMemberFolder(unittest.TestCase):
layer = MY_FUNCTIONAL_TESTING
def setUp(self):
portal = self.portal = self.layer['portal']
def test_members_folder(self):
membership = getToolByName(self.portal, 'portal_membership')
membership.addMember("basicuser", "secret", ["Member"], [])
transaction.commit()
login(self.portal, "basicuser")
# This works just fine, because it was set by my product
self.assertEquals(membership.memberareaCreationFlag, 1,
"memberareaCreationFlag must be 1 when it is enabled")
members_folder = membership.getMembersFolder()
# But this fails
self.assertIsNotNone(members_folder)
# Also we should have the user folder here
self.assertTrue(members_folder.hasObject('basicuser'))
I specifically need Member folder functionality. Just a folder owned by the test user does not cut it.
Also I tried creating new user with acl_users.userFolderAddUser, but that does not help neighter.
The memberareaCreationFlag works just fine in live Plone site.

I finally figured it out.
At first membership.memberareaCreationFlag = 1 is not enough for enabling member folders.
It must be enabled with SecurityControlPanelAdapter in plone.app.controlpanel.security
from plone.app.controlpanel.security import ISecuritySchema
# Fetch the adapter
security_adapter = ISecuritySchema(portal)
security_adapter.set_enable_user_folders(True)
Also the Functional testing fixture does not create the member folder automatically, but is possible to install it manually in your fixture class
class YourPloneFixture(PloneSandboxLayer):
defaultBases = (PLONE_FIXTURE,)
def setUpZope(self, app, configurationContext):
# Required by Products.CMFPlone:plone-content
z2.installProduct(app, 'Products.PythonScripts')
def setUpPloneSite(self, portal):
# Installs all the Plone stuff. Workflows etc.
self.applyProfile(portal, 'Products.CMFPlone:plone')
# Install portal content. Including the Members folder!
self.applyProfile(portal, 'Products.CMFPlone:plone-content')
Finally as Member folders are created uppon user login, but the login helper function in plone.app.testing seem to be too low level for this. Login with zope.testbrowser seems to do the trick
browser = Browser(self.layer['app'])
browser.open(self.portal.absolute_url() + '/login_form')
browser.getControl(name='__ac_name').value = TEST_USER_NAME
browser.getControl(name='__ac_password').value = TEST_USER_PASSWOR
browser.getControl(name='submit').click()
Phew.

self.assert_ isn't a testing method, use something like self.assertTrue, or self.assertIsNotNone.
To add members folders just turn on member folder creation and add a new user.

Something like
def setUpPloneSite(self, portal):
# Install into Plone site using portal_setup
quickInstallProduct(portal, 'Products.DataGridField')
quickInstallProduct(portal, 'Products.ATVocabularyManager')
quickInstallProduct(portal, 'Products.MasterSelectWidget')
if HAVE_LP:
quickInstallProduct(portal, 'Products.LinguaPlone')
applyProfile(portal, 'vs.org:default')
portal.acl_users.userFolderAddUser('god', 'dummy', ['Manager'], [])
setRoles(portal, 'god', ['Manager'])
login(portal, 'god')
is perfectly working for us.

Substitutions inside links in reST / Sphinx

I am using Sphinx to document a webservice that will be deployed in different servers. The documentation is full of URL examples for the user to click and they should just work. My problem is that the host, port and deployment root will vary and the documentation will have to be re-generated for every deployment.
I tried defining substitutions like this:
|base_url|/path
.. |base_url| replace:: http://localhost:8080
But the generated HTML is not what I want (doesn't include "/path" in the generated link):
http://localhost:8080/path
Does anybody know how to work around this?

New in Sphinx v1.0:
sphinx.ext.extlinks – Markup to shorten external links
https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html
The extension adds one new config value:
extlinks
This config value must be a dictionary of external sites, mapping unique short alias names to a base URL and a prefix. For example, to create an alias for the above mentioned issues, you would add
extlinks = {'issue':
('http://bitbucket.org/birkenfeld/sphinx/issue/%s', 'issue ')}
Now, you can use the alias name as a new role, e.g. :issue:`123`. This then inserts a link to http://bitbucket.org/birkenfeld/sphinx/issue/123. As you can see, the target given in the role is substituted in the base URL in the place of %s.
The link caption depends on the second item in the tuple, the prefix:
If the prefix is None, the link caption is the full URL.
If the prefix is the empty string, the link caption is the partial URL given in the role content (123 in this case.)
If the prefix is a non-empty string, the link caption is the partial URL, prepended by the prefix – in the above example, the link caption would be issue 123.
You can also use the usual “explicit title” syntax supported by other roles that generate links, i.e. :issue:`this issue <123>`. In this case, the prefix is not relevant.

I had a similar problem where I needed to substitute also URLs in image targets.
The extlinks do not expand when used as a value of image :target: attribute.
Eventually I wrote a custom sphinx transformation that rewrites URLs that start with a given prefix, in my case, http://mybase/. Here is a relevant code for conf.py:
from sphinx.transforms import SphinxTransform
class ReplaceMyBase(SphinxTransform):
default_priority = 750
prefix = 'http://mybase/'
def apply(self):
from docutils.nodes import reference, Text
baseref = lambda o: (
isinstance(o, reference) and
o.get('refuri', '').startswith(self.prefix))
basetext = lambda o: (
isinstance(o, Text) and o.startswith(self.prefix))
base = self.config.mybase.rstrip('/') + '/'
for node in self.document.traverse(baseref):
target = node['refuri'].replace(self.prefix, base, 1)
node.replace_attr('refuri', target)
for t in node.traverse(basetext):
t1 = Text(t.replace(self.prefix, base, 1), t.rawsource)
t.parent.replace(t, t1)
return
# end of class
def setup(app):
app.add_config_value('mybase', 'https://en.wikipedia.org/wiki', 'env')
app.add_transform(ReplaceMyBase)
return
This expands the following rst source to point to English wikipedia.
When conf.py sets mybase="https://es.wikipedia.org/wiki" the links would point to the Spanish wiki.
* inline link http://mybase/Helianthus
* `link with text <http://mybase/Helianthus>`_
* `link with separate definition`_
* image link |flowerimage|
.. _link with separate definition: http://mybase/Helianthus
.. |flowerimage| image:: https://upload.wikimedia.org/wikipedia/commons/f/f1/Tournesol.png
:target: http://mybase/Helianthus

Ok, here's how I did it. First, apilinks.py (the Sphinx extension):
from docutils import nodes, utils
def setup(app):
def api_link_role(role, rawtext, text, lineno, inliner, options={},
content=[]):
ref = app.config.apilinks_base + text
node = nodes.reference(rawtext, utils.unescape(ref), refuri=ref,
**options)
return [node], []
app.add_config_value('apilinks_base', 'http://localhost/', False)
app.add_role('apilink', api_link_role)
Now, in conf.py, add 'apilinks' to the extensions list and set an appropriate value for 'apilinks_base' (otherwise, it will default to 'http://localhost/'). My file looks like this:
extensions = ['sphinx.ext.autodoc', 'apilinks']
# lots of other stuff
apilinks_base = 'http://host:88/base/'
Usage:
:apilink:`path`
Output:
http://host:88/base/path

You can write a Sphinx extension that creates a role like
:apilink:`path`
and generates the link from that. I never did this, so I can't help more than giving this pointer, sorry. You should try to look at how the various roles are implemented. Many are very similar to what you need, I think.