In my use case, I was monitoring around 200 stocks using client API and getting regular data every 2 mins. I have not renewed my subscription and have decided to get the information from the browser using Python.
The data is available as a CSV download for each stock. Unfortunately the python script takes 0.5 seconds to navigate and retrieve data for one stock before moving onto the next one. This takes 100 seconds in total which is a long time. Windows has an option to create "New Desktops", I would like to create 200 such desktop for each stock through automation and have the data pulled in 0.5 seconds through individual scripts in one go for stocks, but I have not been able to find any resources on creating multiple mouse handlers for each desktop. Since there is only one mouse/keyboard, it cannot be done simultaneously. I cannot use Virtual desktops as there is only one login per client. Is there any programming language that allows us to create virtual mouse so that i can tag one for each desktop? I would appreciate any feedback to further pursue this.
Related
I created a script with python that extracts a KPI from a database and compares it with another KPI, then I created a push notification that sent me a message on the desktop if the KPI of the database is out of its tolerance. My question is how can I automate this push notifier in my desktop? I mean every time the KPI changes I want to receive automatically a message from the notifier on the Desktop? Do you have any idea, please? Thank you :)
If you want to do that locally. I can think of 2 options.
You need to create another function/script that detects if there is a change in the database. Then
You have that script running in the background constantly. You can make it check for changes every minute or so. If there is a change, then it calls the script that you made. You can also make it being executed every time you turn on your PC
Similar to before but with the use of a windows scheduler. That schedules can run the script every minute or so.
I am making a python bot that makes twitter accounts, just for educational purposes with throw-away emails (catchalls) and russian phonenumbers. I managed to get through both email and phone verification and was wondering if I can create accounts at a large scale by starting N webdrivers at once.
Right now I made a code that only loops the program N times. I removed the code but it looked like this:
amount = input....
for i in range(amount)
App.run()
This was my only hope in actually doing this. Does anyone know how I can do this and if a computer can actually handle 10 or 100 headless webdrivers from selenium at once?
Well, you need to create multiple threads instead of looping, then you can start each upload in parallel threads. You are on the right track. You don't need a selenium grid to achieve this.
lookup about multithreading. You can start with this answer(Threads in Java)
It's not right you need grid for executing multiple browser sessions. You can invoke multiple browser sessions by just creating multiple driver objects, and managing them. Each session will be separate if you want them to be.
Grid is for scaling as there is a limitation on the no of browser instances you can run keeping your machine performance intact and tests stable. Like more than 5 chrome instances in a single machine. If you want to do more than that then you have to use selenium Grid.
I am working on a web scraping project using python and an API
I want the python script to be ran everyday for 5 days for 12 hours as a job
I don't want to keep my system alive to either do it in CMD or in Jupyter so I was looking for a solution wherein any cloud service would help me automate the process
One way to do this would be to write a web scraper in Python, and run it on an AWS Lambda, which is essentially a serverless function with no underlying ops to manage. Depending on your use case, you could either perform some action based on the contents of that page data, or you could write the result out to S3 as a file.
To have your function execute in a recurring fashion, you can then set your AWS Lambda event trigger to be a CloudWatch event (in this case, some recurring timer at whatever frequencies/times you'd like, such as once each hour for a 12 hour window during Mon-Fri).
This is typically going to be an easier approach when compared to spinning up a virtual server (EC2 instance), and managing a persistent process that could error during waits/operation for any number of reasons.
Using the webinterface of console.cloud.google.com to delete one Compute Engine VM instance triggers roughly 5 compute.instances.listReferrers requests per second for roughly 5 minutes. These spikes in api usage can be observed under "APIs & Services" -> "Metrics" (change to 1 hour, to improve visibility). I'm looking for a way to not trigger this many requests, for those can easily lock me out of the "2k requests per 100 seconds" quota.
VM specs:
region us-east1 zone us-east1-b
8vCPUs, 7.2GB memory
Ubuntu 18.04 LTS Minimal
preemtibility ON
Deleting the VM instance under "Compute Engine" -> "VM instances" -> three stacked dots on the right side of the vm row -> "Delete" -> "Delete an Instane, Are you sure that you want to delte instance 'X'? (This will also delete boot disk 'X')" -> "Delete"
Viewing activies in a different tab under "APIs & Services" -> "Metrics" (change to 1 hour, to improve visibility) shows a spike in activiy of compute.instances.listReferrers for several minutes afterwards.
I'm located in EU if that matters for this issue.
I expected to see a very short spike in activity, due to collection of resources that need to be deleted. But I didn't expect such a long duration of several minutes. If I delte multiple instances at the same time or in short succession, I'm breaking the allowed quota and can't create new instances for some time.
Finally, if someone knows a way of how this could be brought to the attention of the responsible google devs, please let me know too.
I wrote a little python Script that fetches data from some website using hard coded credentials ( i know its bad but not part of this question).
The website has new data every day and im gathering data from a whole week and parse it into a single .pdf.
I've already adjusted the script to always generate a pdf off last week by default. (no params needed)
Im kinda lazy and don't want to run the script every week by hand.
So is it possible to run the script at certain times, for example every monday at 10am?
Sure, just utilize Windows' task scheduler. There you can create new tasks to your delight and let it run commands to whatever times or intervalls you want. The task schedulers' GUI should be self-explanatory, but to be concrete on your example:
Configure the run time (weekly, monday, 10am) under triggers
Add a new action and give it your Python interpreter as the command and your script to be run as the argument
Configure the rest according to your needs