I would like to read chrome's js console using Python3 without any webdriver such as selenium (bot detection and stuff).
I've tried Chrome DevTools Protocol python libraries such as chromewhip, pychrome and PyChromeDevTools, but I'm unable to read any data from the console.
I want to read Runtime.consoleAPICalled or Log.entryAdded, but I don't know how to implement these callbacks as the documentation for these libraries doesn't specify any of that. Also there are no examples to be found either.
Does anyone know how to properly access these events or some other library which provides it?
#kundapanda could you at least post a snippet of the code that worked for you.
I want to read Runtime.consoleAPICalled or Log.entryAdded, but I don't
know how to implement these callbacks
The following assumes (per your question phrasing) that you're able to send and receive debug protocol messages on the stream that's open to the web debugger endpoint.
After you send debug protocol messages Runtime.enable and Log.enable messages, the Runtime.consoleAPICalled and Log.entryAdded "events" you are looking for are represented by messages you receive on the same debugging channel.
You may need to match the console event messages with the execution context (seen in the Runtime.enable response) by examining the executionContextId field in the received event messages. The log events are not associated with any single execution context. All of these "event" messages will have Id=0, which helps to recognize they're "event" messages and not response messages.
Here are a couple of sample messages received from Chrome (formatted as JSON with arbitrary field order):
Console API event message:
{
"method": "Runtime.consoleAPICalled",
"params":
{
"type": "warning",
"args": [
{ "type": "string",
"value": "Google Maps JavaScript API warning: NoApiKeys https://developers.google.com/maps/documentation/javascript/error-messages#no-api-keys"
}],
"executionContextId": 1,
"timestamp": 1618949706735.553,
"stackTrace":
{
"callFrames": [
{
"functionName": "TA.j",
"scriptId": "206",
"url": "https://maps.googleapis.com/maps-api-v3/api/js/44/10/util.js",
"lineNumber": 228,
"columnNumber": 26
} ]
}
}
},
"id": 0
}
Log event message:
{
"method":"Log.entryAdded",
"params":
{
"entry":
{
"source":"javascript",
"level":"warning",
"text":"The deviceorientation events are blocked by permissions policy. See https://github.com/w3c/webappsec-permissions-policy/blob/master/features.md#sensor-features",
"timestamp":1.6189509536801208e+12
}
},
"id": 0
}
Related
I am using Azure function service bus trigger in Python to receive messages in batch from a service bus queue. Even though this process is not well documented in Python, but I managed to enable the batch processing by following the below Github PR.
https://github.com/Azure/azure-functions-python-library/pull/73
Here is the sample code I am using -
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"cardinality": "many",
"queueName": "<some queue name>",
"dataType": "binary",
"connection": "SERVICE_BUS_CONNECTION"
}
]
}
__init__.py
import logging
import azure.functions as func
from typing import List
def main(msg: List[func.ServiceBusMessage]):
message_length = len(msg)
if message_length > 1:
logging.warn('Handling multiple requests')
for m in msg:
#some call to external web api
host.json
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.3.0, 4.0.0)"
},
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"batchOptions": {
"maxMessageCount": 100,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
}
}
After using this code , I can see that service bus trigger is picking up messages in a batch of 100 (or sometimes < 100) based on the maxMessageCount but I have also observed that most of the messages are ending up in the dead letter queue with the MaxDeliveryCountExceeded reason code. I have tried with different values of MaxDeliveryCount from 10-20 but I had the same result. So my question is do we need to adjust/optimize the MaxDeliveryCount in case of batch processing of service bus messages ? How both of them are related ? What kind of change can be done in the configuration to avoid this dead letter issue ?
From what we discussed in the comments, this is what you encounter:
Your function app is fetching 100 messages from ServiceBus (prefetchCount) and locking them for a maximum of maxAutoRenewDuration
Your function code is processing messages one at a time at a slow rate because of the API you call.
By the time you finish a batch of messages (maxMessageCount), the lock already expired which is why you have exceptions and the message gets redelivered again. This eventually causes MaxDeliveryCountExceeded errors.
What can you do to improve this?
Reduce maxMessageCount and prefetchCount
Increase maxAutoRenewDuration
Increase the performance of your API (how to do that would be a different question)
Your current code would be much better off by using a "normal" single message trigger instead of the batch trigger
PS: Beware that your function app may scale horizontally if you are running in a consumption plan, further increasing the load on your struggling API.
i'm trying to write my own custom webdriver interface to control firefox through chrome devtools protocol (cdp). i launch firefox using firefox-esr --remote-debugging-port 0 and then it creates a websocket that i connect to using python:
async def main_client():
async with websockets.connect("ws://localhost:34805/devtools/browser/f67014fd-9397-478d-a11e-66c189704ab0") as client_connection:
while True:
message = input("type in a command: ")
await client_connection.send(message)
asyncio.run(main_client())
then i send a message in json format
{
"command":"Runtime.evaluate",
"parameters":{"expression": "console.log(\"this is a message\")"}
}
the problem is that when i send the message nothing happens on the receiving end. does anyone know how to send messages over websocket to firefox?
thanks
so i did a bit more research and it turns out that to be able to interact with firefox cdp after starting using ./firefox --remote-debugging-port 12345 you need to make a get request to localhost:12345/json/list. here you will find the list of websocket targets you can connect to. by default you have a top most browser target that doesn't have any tab elements to interact with and then you have tab targets that you can interact with. here is any example of a json list when starting firefox cdp:
[
{
"description": "",
"devtoolsFrontendUrl": null,
"faviconUrl": "",
"id": "ef9b04c6-409f-4fe9-bea9-c50979049820",
"type": "page",
"url": "about:blank",
"webSocketDebuggerUrl": "ws://127.0.0.1:12345/devtools/page/ef9b04c6-409f-4fe9-bea9-c50979049820"
},
{
"description": "Main process target",
"devtoolsFrontendUrl": "",
"faviconUrl": "",
"id": "c726e615-36cc-4a73-a48a-a75cc0fa941e",
"title": "Main process target",
"type": "browser",
"url": "",
"webSocketDebuggerUrl": "ws://127.0.0.1:12345/devtools/browser/c726e615-36cc-4a73-a48a-a75cc0fa941e"
}
]
after getting the json response with the json list then you can connect to one of the websockets using python and then you can send messages through websocket.
not all targets support all cdp commands. also some cdp commands require a response but some don't, although if the command was unsuccessful, you will get a response with the error message.
you can find a list of all cdp commands at https://chromedevtools.github.io/devtools-protocol/ and you can find all the supported cdp commands by your firefox version at http://localhost:12345/json/protocol after launching firefox with cdp.
i still haven't figured out how to run Runtime.evaluate because you need to specify a contextId (it says it's optional but when sending commands through websocket it is required) and i don't know where to get the current context id from. if anyone finds out let me know.
Whilst trying to execute a webhook, I've been getting 404 errors. The url I'm using is exactly what the Discord docs tell me to use: https://discord.com/api/v10/webhooks/{webhook.id}/{webhook.token}. However, I've been able to successfully create and delete webhooks. Finally, to test this error, I waited about a day after creation to see if it took some time for the url to work, but this also turned up a 404 error.
When creating a webhook, I use this method. Upon doing so, it returns json containing something like this:
{
"name": "test webhook",
"type": 1,
"channel_id": "199737254929760256",
"token": "3d89bb7572e0fb30d8128367b3b1b44fecd1726de135cbe28a41f8b2f777c372ba2939e72279b94526ff5d1bd4358d65cf11",
"avatar": null,
"guild_id": "199737254929760256",
"id": "223704706495545344",
"application_id": null,
"user": {
"username": "test",
"discriminator": "7479",
"id": "190320984123768832",
"avatar": "b004ec1740a63ca06ae2e14c5cee11f3",
"public_flags": 131328
}
}
I used this bit of documentation to attempt to execute the webhook to send a message but to no avail. I use a url that uses the id and token in the json of the webhook, so with the JSON above, the url would look like this: https://discord.com/api/v10/webhooks/223704706495545344/3d89bb7572e0fb30d8128367b3b1b44fecd1726de135cbe28a41f8b2f777c372ba2939e72279b94526ff5d1bd4358d65cf11. I have checked to make sure that the program is using the right URL, and it is. What am I doing wrong to get the 404 error?
Thanks in advance.
You are using the API correctly from what I can tell, it's likely something that goes wrong when executing the webhook.
You can verify this by sending a get GET request to your webhook URL (for example by opening it in your browser), and see what response you get.
For example, a successful call for this URL would look like this:
https://discord.com/api/v10/webhooks/730533481892106727/aPKOoEoamLq9pVnKHT3gF-whSrtrsnkWlOA3xPQPMNBsK5vhe9PC82-e7bu3BEFESQfaw
{
"type": 1,
"id": "730533481892106727",
"name": "Spidey Bot",
"avatar": null,
"channel_id": "501452852364050123",
"guild_id": "413797995205889283",
"application_id": null,
"token": "aPKOoEoamLq9pVnKHT3gF-whSrtrsnkWlOA3xPQPMNBsK5vhe9PC82-e7bu3BEFESQfaw"
}
It seems like missing a forward-slash / will result in a 404, so that might be your issue.
My Code is very short but it is not working but in youtube videos it works
import urllib.request
import json
key = "AIzaSyCyTALBjNAEp-Vr5GkilEnLzH0y1Tw6SC4"
data = urllib.request.urlopen("www.googleapis.com/youtube/v3/channels?part=statistics&forUsername=pewdiepie&key="+key).read()
subs = json.loads(data)['items'][0]["statistics"]['subscriberscount']
print(subs)
You need to create you own API key
https://www.googleapis.com/youtube/v3/channels?part=statistics&forUsername=pewdiepie&key=AIzaSyCyTALBjNAEp-Vr5GkilEnLzH0y1Tw6SC4
shows
"error": {
"code": 403,
"message": "YouTube Data API v3 has not been used in project 195563281908 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/youtube.googleapis.com/overview?project=195563281908 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
"errors": [
{
"message": "YouTube Data API v3 has not been used in project 195563281908 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/youtube.googleapis.com/overview?project=195563281908 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
"domain": "usageLimits",
"reason": "accessNotConfigured",
"extendedHelp": "https://console.developers.google.com"
}
],
"status": "PERMISSION_DENIED"
}
}
Here you can find how to create own API key:
https://developers.google.com/youtube/v3/getting-started
I have a slack /Slash command that generates the app file link. I want to send that link to a user as a button or link but it has to be prefixed with:
itms-services://?action=download-manifest&url={download_link}
Currently slack is not recognizing this as a valid link but it's mandatory by iOS because of permission issues.
Basically I want to mimic the download button on web on slack so user is not required to visit the website.
On Web this link itms-services://?action=download-manifest&url={download_link} works fine and user is asked to authorize the download.
test_attachment = [
{
"color": "#CC0000",
"actions": [
{
"type": "button",
"text": ":red_circle: Download Link:",
"url": "itms-services://?action=download-manifest&url={download_link}"
}
]
}
]
slack_client.api_call("chat.postMessage", channel=channel_id, text=text, attachments=json.dumps(test_attachment))
Another example:
<a href="itms-services://?action=download-manifest&url=http://loqi.me/app/Geoloqi.plist">
Download Geoloqi
</a>
This will work on Web and how to make it work on slack messages.
An itms-services link is not recognize as a valid URL by slack parser because its address (the thing we usually found just after ://) is empty.
A quick fix is to fill it with a dummy address for example 0.0.0.0.
Here is a block kit that works :
{
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Try to install me:"
},
"accessory": {
"type": "button",
"text": {
"type": "plain_text",
"text": "Install",
"emoji": true
},
"url": "itms-services://0.0.0.0?action=download-manifest&url={download_link}"
}
}
]
}
and it looks like :
I had a similar problem in one of my apps where I wanted to provide a download button for a dynamically generated image file.
As far as I know there is no switch or option in Slack that allows you to do this, so your only option is to use a helper web app.
The approach would be:
User clicks a link button on Slack that opens your helper web app in the browser
Helper web app performs the actual download (no further user interaction required)
My guess is the user will need to authenticate in the browser anyways, so this should not create any inconvenience for the user. But of course you would need to provide the additional web helper app.