I am working on an application that sends logs to GCP StackDriver. I want to put custom "tags" (or summary fields) natively on my log entry. I am looking for a solution that doesn't rely on defining custom summary fields in the console, as those are not permanent, and not project-wide.
I realized that some logger have tags displayed. For example, GCF logs will show its execution_id. Using the following snippet, I can verify that the tags displayed depend on the name of the logger:
from google.cloud import logging
client = logging.Client()
client.logger(name="custom").log_text("foobar", labels={"execution_id": "foo"})
client.logger(name="cloudfunctions.googleapis.com%2Fcloud-functions").log_text("foobar", labels={"execution_id": "foo"})
if you filter your logs on "foobar", you will see that only the second entry has "foo" as a tag.
That tag matches the label.execution_id specified in the code. The problem is, I cannot add custom labels, if I add another label that is not execution_id, it is not displayed as a tag (but still found in the log body).
It looks like each monitored resources has its own set of tag, ie: BigQuery resources use protoPayload.authenticationInfo.principalEmail as tag. But I cannot find a way to specify my own resources.
Does anybody has some experience with that kind of issue?
Thanks in advance
The closest solution I found was in an expanded log entry, click on a field within the JSON representation. In the resulting panel, select Add field to summary line,
to get more information about this topic, please refer to this link
Additionally I found a feature request opened for the product team, where the user, on that case, wants to filter out in Stackdriver by Dataflow jobs custom labels, the reference might be useful on your use case, no ETA was shared, neither guarantee of its implementation
I've filed a Feature Request on your behalf to the product team, they'll evaluate the possibility to implement the functionality that fits your use case, you can follow up on this PIT [1], where you will be able to receive further updates from the team as well.
Keep in mind that there is no ETA, nor guarantee that this will be implemented. However, please feel free to ask for updates directly on the PIT, I would appreciate if you give my answer as accepted, if it was helpful for you.
[1]https://issuetracker.google.com/172667238
Related
I'm using web3py and I want to get the transaction history of a specific contract. Here's my code sample
eventSignatureHash = web3.keccak(text='Transfer(address,uint256)').hex()
filter = web3.eth.filter({
'address': '0x828402Ee788375340A3e36e2Af46CBA11ec2C25e',
'topics': [eventSignatureHash]
})
I'm expected to get ERC20 Token Transactions related to this contract as found here but it does not display anything unfortunately. How to go about this?
Finally, is there a way to watch these transactions in real time?
What i did is that i created a contract instance:
contract = web3.eth.contract(address = contract_address)
then trasnfer_filter = contract.events.Transfer.filter(u have optional parameters such as: fromBlock:...,toBlock, argument_filters:{"to": users_address(this filters for transfers only to that address)})
so you can play around it.
https://web3py.readthedocs.io/en/latest/contracts.html#web3.contract.ContractEvents
found in the event log object section.
As is answered by other's answer, contract.events provides lots of useful methods. If nothing is returned, specifying from and to might help.
And besides, an ultimate solution is already provided here -> Advanced example: Fetching all token transfer events.
Finally, is there a way to watch these transactions in real time?
Actually, lots of nodes support subscribe RPCs. However, this feature is not yet supported by web3py(#1402). You can try SDK in other language or temporarily adopt this method here
How do I set custom "trace_id" for Datadog tracing? I searched high and low but can't find an answer to this. I suspect it's not supported. Would really appreciate it if I can get some help here.
As an example, if I can do the following in multiple files, then I can view these spans together in the Datadog UI since they all have the same trace ID:
#tracer.wrap(service='foo', resource='bar')
def bar(self, ttt):
span = tracer.current_span()
span.set_trace_id("my_customer_trace_id")
It turns out that trace id can be set via HTTP endpoint https://docs.datadoghq.com/api/v1/tracing/#send-traces. There doesn't seem to be an option for sending traces to the agent directly.
This can still be useful if the performance penalty of making HTTP calls is not a concern, i.e., if you are not working on a real-time system.
I am not well familiar with Datadog UI, but I see that ddtrace allow you to set tags:
span.set_tag('your_own_id', '12345')
So we have been using Apache Superset, It's a great tool.
The only frustration come from that there are a few dashboards we want to share with users outside the company.
I believe right now the way to do it is go from the Gamma user then create a read only role (Correct me if I'm wrong)
There are a few downside of this:
we need to create a view per user on each table to make sure that they do not see the records that they are not supposed to.
the access is given by datasource, so they will be able to see any dashboard that use the same datasource, which can be a problem sometimes.
all of these authentication is a lot of work to maintain.
I'm wondering if there is any way (or even hack) to simply share the graphs and tables as a dashboard, without any database access granted.
Like a frozen or snapshot of dashboard,
like the way Redash does it:
https://redash.io/help/user-guide/dashboards/sharing-dashboards
What you are looking for can be achieved through a combination of the public user and appending ?standalone=true to the dashboard url.
You also don't need the entire list of Gamma user permissions, the most important ones are can explore on superset, explore json on superset and datasource access and csrf token. This basically renders the dashboards without the superset menu and should make everything readonly.
We can achieve this by creating a custom role.
1. Can remove all the menu items
2. Can disable the dashboard edit button
3. Can give access to specific tables.
So a user cannot access any other dashboard or charts
Eg. Dashboard
Public dashboards
This is not meant for production. It’s for experiments or while doing a proof of concept.
#superset_config.py
PUBLIC_ROLE_LIKE_GAMMA = True
or
PUBLIC_ROLE_LIKE: Optional[str] = "Gamma"
After this, we need to re-run the init user (if already run)
docker-compose exec superset superset-init
Dashboards & charts can be embedded without superset header (Nav bar etc) by adding standalone=true parameter to the url, like this :
http://localhost:9000/superset/dashboard/world_health/?standalone=true
We need to grant database source permissions to public role for the data to be visible.
please see: https://sairamkrish.medium.com/apache-superset-custom-authentication-and-integrate-with-other-micro-services-8217956273c1
Superset is great, I'm glad people are talking about it since the days when it was AirBnB's Caravel. It has come a long way.
There is no 'official' solution for what you're looking for but there is a way to effectively get the same result. You said you wouldn't mind a 'hack' so...
Creating a table or a data source and exposing it to the 'public' group should do what you're looking to accomplish.
I've looked around for a little while now and can't seem to find anything that even touches on the differences. As the title states, I'm trying to find out what difference getting your data via url path parameters like /content/7 then using regex in your urls.py, and getting them from query params like /content?num=7 using request.GET.get() actually makes.
What are the pros and cons of each, and are there any scenarios where one would clearly be a better choice than the other?
Also, from what I can tell, the (Django's) preferred method seems to be using url path params with regex. Is there any reason for this, other than potentially cleaner URLs? Any additional information pertinent to the topic is welcome.
This would depend on what architectural pattern you would like to adhere to. For example, according to the REST architectural pattern (which we can argue is the most common), you want do design URLs such that without query params, they point to "resources" which roughly correspond to nouns in your application and then HTTP verbs correspond to actions you can perform on that resource.
If, for instance, your application has users, you would want to design URLs like this:
GET /users/ # gets all users
POST /users/ # creates a new user
GET /users/<id>/ # gets a user with that id. Notice this url still points to a user resource
PUT /users/<id> # updates an existing user's information
DELETE /users/<id> # deletes a user
You could then use query params to filter a set of users at a resource. For example, to get users that are active, your URL would look something like
/users?active=true
So to summarize, query params vs. path params depends on your architectural preference.
A more detailed explanation of REST: http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api
Roy Fielding's version if you want to get really academic: http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm
I have a simple GAE system that contains models for Account, Project and Transaction.
I am using Django to generate a web page that has a list of Projects in a table that belong to a given Account and I want to create a link to each project's details page. I am generating a link that converts the Project's key to string and includes that in the link to make it easy to lookup the Project object. This gives a link that looks like this:
My Project Name
Is it secure to create links like this? Is there a better way? It feels like a bad way to keep context.
The key string shows up in the linked page and is ugly. Is there a way to avoid showing it?
Thanks.
There is few examples, in GAE docs, that uses same approach, and also Key are using characters safe for including in URLs. So, probably, there is no problem.
BTW, I prefer to use numeric ID (obj_key.id()), when my model uses number as identifier, just because it's looks not so ugly.
Whether or not this is 'secure' depends on what you mean by that, and how you implement your app. Let's back off a bit and see exactly what's stored in a Key object. Take your key, go to shell.appspot.com, and enter the following:
db.Key(your_key)
this returns something like the following:
datastore_types.Key.from_path(u'TestKind', 1234, _app=u'shell')
As you can see, the key contains the App ID, the kind name, and the ID or name (along with the kind/id pairs of any parent entities - in this case, none). Nothing here you should be particularly concerned about concealing, so there shouldn't be any significant risk of information leakage here.
You mention as a concern that users could guess other URLs - that's certainly possible, since they could decode the key, modify the ID or name, and re-encode the key. If your security model relies on them not guessing other URLs, though, you might want to do one of a couple of things:
Reconsider your app's security model. You shouldn't rely on 'secret URLs' for any degree of real security if you can avoid it.
Use a key name, and set it to a long, random string that users will not be able to guess.
A final concern is what else users could modify. If you handle keys by passing them to db.get, the user could change the kind name, and cause you to fetch a different entity kind to that which you intended. If that entity kind happens to have similarly named fields, you might do things to the entity (such as revealing data from it) that you did not intend. You can avoid this by passing the key to YourModel.get instead, which will check the key is of the correct kind before fetching it.
All this said, though, a better approach is to pass the key ID or name around. You can extract this by calling .id() on the key object (for an ID - .name() if you're using key names), and you can reconstruct the original key with db.Key.from_path('kind_name', id) - or just fetch the entity directly with YourModel.get_by_id.
After doing some more research, I think I can now answer my own question. I wanted to know if using GAE keys or ids was inherently unsafe.
It is, in fact, unsafe without some additional code, since a user could modify URLs in the returned webpage or visit URL that they build manually. This would potentially let an authenticated user edit another user's data just by changing a key Id in a URL.
So for every resource that you allow access to, you need to ensure that the currently authenticated user has the right to be accessing it in the way they are attempting.
This involves writing extra queries for each operation, since it seems there is no built-in way to just say "Users only have access to objects that are owned by them".
I know this is an old post, but i want to clarify one thing. Sometimes you NEED to work with KEYs.
When you have an entity with a #Parent relationship, you cant get it by its ID, you need to use the whole KEY to get it back form the Datastore. In these cases you need to work with the KEY all the time if you want to retrieve your entity.
They aren't simply increasing; I only have 10 entries in my Datastore and I've already reached 7001.
As long as there is some form of protection so users can't simply guess them, there is no reason not to do it.