I need an help to create a python watchdog system to monitor a GitHub repo.
The idea is: every time something new is pushed on a repo (master), I'd like to update my repo on a server with the new commits/push.
Any idea how can do this?
OK! I've found 2 solutions for this
1. If you own the repo that you want to watch
In you're repo page, Go to settings
click webhooks, new webhook (top right)
give it your ip/endpoint and setup everything to your liking
use any server (such as in #pigeonhands answer) to get notified
2. Not your Repo
take the url you want i.e https://github.com/fire17/gd-xo/
add /commits/master.atom to the end such as:
https://github.com/fire17/gd-xo/commits/master.atom
Use any library you want to get that page's content, like:
import requests;
response = requests.get("http://api.open-notify.org/astros.json").text
filter out the keys you want, for example the element
>>> response.split("<updated>")[1].split("</updated>")[0]
'2021-08-06T19:01:53Z'
make a loop that checks this every so often and if this string has changed, then you can initiate a clone/pull request or do whatever you like :)
I'm so happy to finally achieve this capability
Hope this helped!
One solution is to use the github 'push' webhook.
This means instead of continually polling the repo for changes, you can setup a web service where github can tell you when a push has occurred.
A very basic example with flask might look like this
from flask import Flask, request, abort
import hmac
app = Flask(__name__)
#app.route("/myhook", methods=['POST'])
def github_push_webhook():
signature = request.headers.get("X-Hub-Signature")
digest = hmac.new(GITHUB_SECRET.encode(), request.data, hashlib.sha1).hexdigest()
if not hmac.compare_digest(signature, "sha1=" + digest):
abort(400, "Bad Request")
webhook_data = request.get_json()
changed_repository = webhook_data.get('repository')
# clone the repoisitory...
if you own the repo the best way is via webhooks as #pigeonhands suggested, otherwise either a scraper or rss feed combined with telegram/discord/slack webhooks
reference: https://github.com/orgs/community/discussions/22956
Related
Hi i have a flask application that Build as a docker image to serve as an api
this image is deployed to multiple environments (DEV/QA/PROD)
i want to use an applicationInsight for each environment
using a single application Insight works fine
here is a code snippet
app.config['APPINSIGHTS_INSTRUMENTATIONKEY'] = APPINSIGHTS_INSTRUMENTATIONKEY
appinsights = AppInsights(app)
#app.after_request
def after_request(response):
appinsights.flush()
return response
but to have multiple application i need to configure app.config with the key of the app insight
i thought of this solution which thourghs errors
here a snippet :
app = Flask(__name__)
def monitor(key):
app.config['APPINSIGHTS_INSTRUMENTATIONKEY'] = key
appinsights = AppInsights(app)
#app.after_request
def after_request(response):
appinsights.flush()
return response
#app.route("/")
def hello():
hostname = urlparse(request.base_url).hostname
print(hostname)
if hostname == "dev url":
print('Dev')
monitor('3ed57a90-********')
if hostname == "prod url":
print('prod')
monitor('941caeca-********-******')
return "hello"
this example contains the function monitor which reads the url and decide which app key to give so it can send metrics to the right place but apparently i can't do those processes after the request is sent (is there a way a config variable can be changed based on the url condition ?)
error Message :
AssertionError: The setup method 'errorhandler' can no longer be called on the application. It has already handled its first request, any changes will not be applied consistently. Make sure all imports, decorators, functions, etc. needed to set up the application are done before running it.
i hope someone can guide me to a better solution
thanks in advance
AFAIK, Normally Application Insights SDK collect the telemetry data, and it has sent to Azure by batch. So, you have to keep a single application insights resource for an application. Use staging for use different application insights for same application.
When the request started for the service till to complete his response the Application insights has taking care of the specific service life cycle. The application while start to end it will track the information. So that we can't use more than one Application Insights in single application.
When Application starts the AI start collecting telemetry data when Application stops then the AI stops gathering telemetry information. We were using Flush to even though in between application stops to send information to AI.
I have tried what you have used. It confirms the same in the log
I have tried with single application insights I can be able to collect all telemetry information.
I finally made a project that I wanted to make since a long time :
I'm using an Arduino Uno to replace my PC power button (with a simple relay) and that Arduino Board is connected to a Raspi 3 for network connection purposes
My wish is to do a webpage (or a API-Like request) that at a touch of a button (preferably in a password-protected page) It'll power the PC on
I know how to code in Python, and my script to control the Arduino is already done but I can't find a way to run, only server-side, a Python Script from a button in a webpage
I found that CherryPy framework but I don't think it'll suit my needs
Can someone give me any ideas about that please?
As already mentioned by #ForceBru, you need a python webserver.
If this can be useful to you, this is a possible unsecure implementation using flask:
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route('/turnOn')
def hello_world():
k = request.args.get('key')
if k == "superSecretKey":
# Do something ..
return 'Ok'
else:
return 'Nope'
If you put this in an app.py name file and, after having installed flask (pip install flask), you run flask run you should be able to see Ok if visiting the url http://localhost:5000/turnOn?key=superSecretKey .
You could write a brief html gui with a button and a key field in a form but I leaves that to you (you need to have fun too!).
To avoid potential security issues you could use a POST method and https.
Look at the flask documentation for more infos.
My goal is to setup a buildbot that listens out for webhooks from a github server and then builds the repo listed in the webhook via a generic make all command.
The issue I'm having is that it appears in the build steps I need to specify the github repo in advance i.e.
factory.addStep(
steps.GitHub(
repourl= "github.<domain>.com/<user>/<repo>/",
mode='full',
method='clobber'
)
)
Where as ideally I'd want to be able to grab the repo url from the http request (obviously validating it before blindly running code) and then check it out. Something like:
factory.addStep(
steps.GitHub(
repourl= request["repo_url"],
mode='full',
method='clobber'
)
)
Is this possible in the buildbot framework? Any tips or additional documentation to look at would be greatly appreciated!!
Just encase anyone else comes across this I found two potential solutions. First there is an undocumented option in the webhook which allows to add all of the HTTP request information into the properties object:
'www' : {
...
"change_hook_dialects" : {'github': {"github_property_whitelist": "*"}},
...
}
This then gives you access to all the http request info in the scheduler / builder stages. You can then also grab property information in the build_steps stage using the util properties i.e.
factory.addStep(
steps.GitHub(
repourl= util.Property('repository'),
mode='full',
method='clobber',
submodules=True
)
)
I currently have a Flask web server that pulls data from a JSON API using the built-in requests object.
For example:
def get_data():
response = requests.get("http://myhost/jsonapi")
...
return response
#main.route("/", methods=["GET"])
def index():
return render_template("index.html", response=response)
The issue here is that naturally the GET method is only run once, the first time get_data is called. In order to refresh the data, I have to stop and restart the Flask wsgi server. I've tried wrapping various parts of the code in a while True / sleep loop but this prevents werkzeug from loading the page.
What is the most Pythonic way to dynamically GET the data I want without having to reload the page or restart the server?
You're discussing what are perhaps two different issues.
Let's assume the problem is you're calling the dynamic data source, get_data(), only once and keeping its (static) value in a global response. This one-time-call is not shown, but let's say it's somewhere in your code. Then, if you are willing to refresh the page (/) to get updates, you could then:
#main.route("/", methods=['GET'])
def index():
return render_template("index.html", response=get_data())
This would fetch fresh data on every page load.
Then toward the end of your question, you ask how to "GET the data I want without having to reload the page or restart the server." That is an entirely different issue. You will have to use AJAX or WebSocket requests in your code. There are quite a few tutorials about how to do this (e.g. this one) that you can find through Googling "Flask AJAX." But this will require an JavaScript AJAX call. I recommend finding examples of how this is done through searching "Flask AJAX jQuery" as jQuery will abstract and simplify what you need to do on the client side. Or, if you wish to use WebSockets for lower-latency connection between your web page, that is also possible; search for examples (e.g. like this one).
To add to Jonathan’s comment, you can use frameworks like stimulus or turbo links to do this dynamically, without having to write JavaScript in some cases as the frameworks do a lot of the heavy lifting. https://stimulus.hotwired.dev/handbook/origin
I have a website developed in flask running on an apache2 server that responds on port 80 to two URLs
Url-1 http://www.example.com
Url-2 http://oer.example.com
I want to detect which of the two urls the user is coming in from and adjust what the server does and store the variable in a config variable
app.config['SITE'] = 'OER'
or
app.config['SITE'] = 'WWW'
Looking around on the internet I can find lots of examples using urllib2 the issue is that you need to pass it the url you want to slice and I cant find a way to pull that out as it may change between the two with each request.
I could fork the code and put up two different versions but that's as ugly as a box of frogs.
Thoughts welcome.
Use the Flask request object (from flask import request) and one of the following in your request handler:
hostname = request.environ.get('HTTP_HOST', '')
url = urlparse(request.url)
hostname = url.netloc
This will get e.g. oer.example.com or www.example.com. If there is a port number that will be included too. Keep in mind that this ultimately comes from the client request so "bad" requests might have it set wrong, although hopefully apache wouldn't route those to your app.