I have a working Python automation program combine_excel.py. It accesses a server and extracts excel files and combining them with an automation workflow. Currently, I need to execute this automation manually.
I like to host this program in a cloud/server and activate the script at preset timings and at regular intervals. I like to know if there is any service out there that will allow me to do that. Can I do this on Google Cloud or AWS?
The program will generate an output that I could have it saved into to my Google Drive.
An easy/cost-effective way to achieve this could be to use AWS Lambda functions. Lambda functions can be set to trigger at certain time intervals using CRON syntax.
You might need to make some minor adjustments to match some format requirements Lambda has, maybe workout a way to include dependencies if you have any, but everything should be pretty straightforward as there is a lot of information available on the web.
The same can be achieved using Google Cloud Functions.
You could also try Serverless Framework which would take care of the deployment for you, you only need to set it up once.
Another options is to try Zeit it's quite simple to use, and it has a free tier (as the others).
Some useful links:
https://serverless.com/blog/serverless-python-packaging/
https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
https://cloud.google.com/functions/docs/quickstart-python
https://zeit.co/docs/runtimes#official-runtimes/python
Related
Hi is there anyone who is help me to Integrate BIRT report with Django Projects? or any suggestion for connect third party reporting tools with Django like Crystal or Crystal Clear Report.
Some of the 3rd-party Crystal Reports viewers listed here provide a full command line API, so your python code can preview/export/print reports via subprocess.call()
The resulting process can span anything between an interactive Crystal Report viewer session (user can login, set/change parameters, print, export) and an automated (no user interaction) report printing/exporting.
While this would simplify your code, it would restrict deployment to Windows.
For prototyping, or if you don't mind performance, you can call from BIRT from the command line.
For example, download the POJO runtime and use the script genReport.bat (IIRC) to generate a report to a file (eg. PDF format). You can specify the output options and the report parameters on the command line.
However, the BIRT startup is heavy overhead (several seconds).
For achieving reasonable performance, it is much better to perform this only once.
To achieve this goal, there are at least two possible ways:
You can use the BIRT viewer servlet (which is included as a WAR file with the POJO runtime). So you start the servlet with a web server, then you use HTTP requests to generate reports.
This looks technically old-fashioned (eg. no JSON Requests), but it should work. However, I never used this approach.
The other option is to write your own BIRT server.
In our product, we followed this approach.
You can take the viewer servlet as a template for seeing how this could work.
The basic idea is:
You start one (or possibly more than one) Java process.
The Java process initializes the BIRT runtime (this is what takes some seconds).
After that, the Java process listens for requests somehow (we used a plain socket listener, but of course you could use HTTP or some REST server framework as well).
A request would contain the following information:
which module to run
which output format
report parameters (specific to the module)
possibly other data/metadata, e.g. for authentication
This would create a RunAndRenderTask or separate RunTask and RenderTasks.
Depending on your reports, you might consider returning the resulting output (e.g. PDF) directly as a response, or using an asynchronous approach.
Note that BIRT will happily create several reports at the same time - multi-threading is no problem (except for the initialization), given enough RAM.
Be warned, however, that you will need at least a few days to build a POC for this "create your own server" approach, and probably some weeks for prodction quality.
So if you just want to build something fast to see if the right tool for you, you should start with the command line approach, then the servlet approach and only then, and only if you find that the servlet approach is not quite good enough, you should go the "create your own server" way.
It's a pity that currently there doesn't seem to exist an open-source, production-quality, modern BIRT REST service.
That would make a really good contribution to the BIRT open-source project... (https://github.com/eclipse/birt)
We have made an python client which is used as an interface for user. some function is defined in the client which internally calls the APIs and give output to users.
My requirement is to automate the python client - functions and validate the output.
Please suggest tools to use.
There are several ways to do that:
You can write multiple tests for your application as the test cases which are responsible to call your functions and get the result and validate them. It calls the "feature test". To do that, you can use the python "unittest" library and call the tests periodically.
If you have a web application you can use "selenium" to make automatic test flows. (Also you can run it in a docker container virtually)
The other solution is to write another python application to call your functions or send requests everywhere you want to get the specific data and validate them. (It's the same with the two other solutions with a different implementation)
The most straightforward way is using Python for this, the simplest solution would be a library like pytest. More comprehensive option would be something like Robot framework
Given you have jmeter in your tags I assume that at some point you will want to make a performance test, however it might be easier to use Locust for this as it's pure Python load testing framework.
If you still want to use JMeter it's possible to call Python programs using OS Process Sampler
I need to download my company's marketing campaigns on Bing Ads from Airflow. I have seen there is a Bing Ads SDK in Python that may be a good starting point to build a custom operator that would make an API call to Bing Ads and upload the resulting csv to S3 using the s3 hook. I previously used the Google Ads client library to download campaigns as a list of Python objects inside the PythonOperator, but this library was way more intuitive than BingAds.
There is no BingAds hook in Airflow unfortunately, so I have to rely fully on this Python library and wrap it inside the PythonOperator. Has someone built a similar solution in the past that can point me in the right direction?
Best,
I think it's ok to use PythonOperator to run python code - including importing the Bing API - you just have to make sure that the packages installed for scheduler and worker. No need to build custom operator for that.
You can even use task flow API to integrate it nicely https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
Airlfow is meant to be extended in customized - so writing your Python code is entirely expected. More - if you find it works for you, you might actually to contribute it back to Airflow after adding all the "generalisations" - BingAdsProvider would be nice.
I have a python script on my local machine that reads a CSV file and outputs some metrics. The end goal is to create a web interface where the user uploads the CSV file and the metrics are displayed, while all being hosted on Azure.
I want to use a VM on Azure to run this python script.
The script takes the CSV file and outputs metrics which are stored in CosmosDB.
A web interface reads from this DB and displays graphs from the data generated by the script.
Can someone elaborate on the steps I need to follow to achieve this? Detailed steps are not essentially required, but a brief overview with links to relevant learning sources would be helpful.
There's an article that lists the primary options for hosting sites in Azure: https://learn.microsoft.com/en-us/azure/developer/python/quickstarts-app-hosting
As Sadiq mentioned, Functions is probably your best choice as it will probably be less expensive, less maintenance, and can handle both the script and the web interface. Here is a python tutorial for that method: https://learn.microsoft.com/en-us/azure/developer/python/tutorial-vs-code-serverless-python-01
Option 2 would be to run a traditional website on an App Service plan, with background tasks handled either by Functions or a Webjob- they both use the webjobs SDK, so the code is very similar: https://learn.microsoft.com/en-us/learn/paths/deploy-a-website-with-azure-app-service/
VMs are an option if either of those two don't work, but it comes with significantly more administration. This learning path has info on how to do this. The website is built on the MEAN stack, but is applicable to Python as well: https://learn.microsoft.com/en-us/learn/paths/deploy-a-website-with-azure-virtual-machines/
I am trying to write a python program that will alert me when a VM is down. I know PowerShell might be better but would prefer python.
Why do you think it would be better with PowerShell :). Python rules ;)
If you want a more reactive programming, you should look at EventGrid + LogicApp + WebApp/Function first. It's like IFTTT for Azure, EventGrid will trigger an event, and LogicApp will be able to consume this event and send it to a WebApp or Function (that you can write in Python).
Example:
https://learn.microsoft.com/en-us/azure/event-grid/monitor-virtual-machine-changes-event-grid-logic-app
If you want a more "I pull every minute" experience, just use the azure-mgmt-compute package:
https://pypi.org/project/azure-mgmt-compute/
Basic sample:
https://github.com/Azure-Samples/virtual-machines-python-manage
You will need the instance view of the VM, to get the power state, with instance_view
Hope this helps!
(I work at MS in the Azure SDK for Python team)
EDIT:
It seems EventGrid does not support trigger from VM power state yet, you could still use LogicApp with a poll schedule for solution 1: https://learn.microsoft.com/en-us/azure/connectors/connectors-native-recurrence