I am working on a web automation project, which involves working with the same webpage but different accounts at the same time. For this purpose, I am using firefox multi-account containers.
Can anyone help me with how to automate container operations using python and selenium, eg, open a site in a new tab with specific container?
Related
I am currently trying to get a web scraping script to be used on Google Cloud to import data to the Google Sheets API. I had a quick look around and Google Cloud Run seems that it may be able to do this. Unfortunately the script requires Selenium and headless browsing to run the script and I'm coming up short on how to do this properly. I am quite new to the idea of serverless scripts and can't seem to find much out on the web that's specific enough to my needs with regards to Selenium.
I have been able to Dockerize the script/dependencies with Selenium with headless Chrome. However I'm wondering what may be the best way to deploy this in Google Cloud ?
Any thoughts would be appreciated.
See https://dev.to/googlecloud/using-headless-chrome-with-cloud-run-3fdp for an in-depth example of how to use Headless Chrome + Selenium on Cloud Run.
I made an Amazon Web Scraper and I want it work as an extension. What will happen is that it will display the price if it is lower than the previously recorded price. I don't know Javascript. I went through things like Transcrypt but didn't understand much
You cannot. Chrome extensions are written in JS.
The only way to accomplish what you want is to use the extension as a bridge from users browser to your script. You'll need to convert the script into a server of some kind that can accept requests from the extension and respond.
I'm trying to write a python executable that runs in the background and records the amount of time I spend on facebook. This is one of my first projects in software development. The problem is, I haven't found a way to access the chrome browser and look at what URL's are open.
I've used selenium webdriver, but it seems to only be able to open up new tabs.
I've found some kind of google extension called google tabs api, but i'm not entirely sure if it even is a library I can import into python or if it's even meant for python.
I want to run BeautifulSoup and selenium webdriver in amazon lambda and my running environment is python 3.6. Is it possible to run ? if so How. My intention is to scrap datas from a webpage using beautiful soup 4 and selenium(Since it has to scrap data dynamically generated by javascript).
Yes, it's possible. You need to package a headless Chrome binary and chromedriver along with all the Python packages you need. You'll also need to set several options in Selenium's Chrome web driver to make it work.
I wrote a step-by-step tutorial after spending several frustrating weeks trying to deploy it.
You will need to create a deployment package and upload it to Lambda if you are going to use dependancies outside of the standard library.
I have a write up about using BS4 and Lambda together. I did not use Selenium within Lambda but I do have extensive Selenium experience. You will not be able to execute commands within a browser using Lambda. You are going to need to have a remote server stood up, running Selenium Server. Download Selenium and the webdrivers on the machine that you wish to do the web scraping, start the .jar file, it will open a port on the machine Selenium will communicate with.
Considering that you will need a machine running probably windows to fire up a browser and scrape these pages, you probably don't need lambda in the end.
I have process that uses iMacros in Firefox to open some websites and click on some buttons and do some stuff (not any weird stuff, internal work pages). The problem is that I basically can't use my computer while that happens.
I want to automate this via python and found this:
Integrating iMacros scripts into python
However the answer to that question and the links mention that I need the business or enterprise version of it.
Is there a way to just do something like:
Open firefox (I know how)
Use (as a plugin) iMacros to run a iim script in x location
Thanks!!
You can have 100% control over Firefox with Python, as both are open source. The trick is to figure out details. Here are some starting points
Python can script Firefox with Selenium WebDriver
With some tricks, you can dive deeper into Firefox what basic Selenium interaction offers, like opening a web pages. This would include giving direct commands to plugins. Here is an example of settings Firefox profile in a mode that normal security restrictions do not apply.
You need to study Firefox architecture how you can trigger iMacros plugin commands from Selenium. This is the tricky part as this is very marginal use case and there might not be much information available. Expect spending few days of learning Firefox internals.
My guess is that you can disable Firefox security, and then use Selenium WebDriver to run a JavaScript snippet which gives direct commands to iMacros component.