We Keep Coding

Python Selenium Custom FireFox Profile keep data without discarding them after session

i have a website that i want to login to and it should stay like that for multiple sessions
I tried pickle to save cookies once logged in and then load the cookies when running the script again
but this doesn't work the website logs me out
So i tried to set custom profile for firefox
but checking which profile its running on by adding code
# 2- get tmp file location
profiletmp = driver.firefox_profile.path
# but... the current profile is a copy of the original profile :/
print("running profile " + profiletmp)
shows me that i'm running in a tmp directory copied from the main profile mentioned /tmp/xxxxxxxx/webdriver-py-copy
so according to the answer here (Python / Selenium / Firefox: Can't start firefox with specified profile path) i tried copying the folder to main profile but still when loading browser all changes are lost (ex; i made a bookmark and logged into a site then copied the folder over to main profile replacing all the files but still when opening firefox all changes are lost)
i can only retain changes if i manually use Firefox profile and then login without geckodriver now when gecko driver loads the page i'm logged in
by adding custom profile location in chromedriver it retains data but i have different issues with chrome and i want to use firefox

Automate the DHL CSV dump download using Python

I have a requirement of automating the mail sending process with the data from DHL. Currently what we are doing is:
We have a DHL account, someone has to manually login to the account , download the CSV dump which contains the order tracking details then upload it to the server, port the data from those and process it.
So I thought of automating the whole process so that it requires minimal manual intervention.
1) Is there anyway we can automate the download process from DHL?
Note: I'm using Python

I'd start by looking for something more convenient to access with code...
searching google for "dhl order tracking api" gives:
https://developer.dhl/api-catalog
as its first result, which looks useful and exposes quite a bit of functionality.
you then need to figure out how to make a "RESTful" request, which has answers here like Making a request to a RESTful API using python, and there are lots of tutorials on the internet if you search for things like "python tutorial rest client" which points to articles like this

You can use Selenium for Python. Selenium is a package that automates a browser session. you can simulate mouse clicks and other actions using Selenium.
To Install:
pip install selenium
You will also have to install the webdriver for the browser you prefer to use.
https://www.seleniumhq.org/projects/webdriver/
Make sure that the browser version that you are using is up to date.
Selenium Documentation: https://selenium-python.readthedocs.io/
Since you are dealing with passwords and sensitive data, I am not including the code.

Login and Download
You can automate download process using selenium. Below is the sample code to automate any login process and download items from a webpage. As the requirements are not specific I'm taking general use-case and explaining how to automate the login and download process using python.
# Libraries - selenium for scraping and time for delay
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
chromeOptions = webdriver.ChromeOptions()
prefs = {"download.default_directory" : "Path to the directory to store downloaded files"}
chromeOptions.add_experimental_option("prefs",prefs)
chromedriver = r"Path to the directory where chrome driver is stored"
browser = webdriver.Chrome(executable_path=chromedriver, chrome_options=chromeOptions)
# To maximize the browser window
browser.maximize_window()
# web link for login page
browser.get('login page link')
time.sleep(3) # wait for the page to load
# Enter your user name and password here.
username = "YOUR USER NAME"
password = "YOUR PASSWORD"
# username send
# you can find xpath to the element in developer option of the chrome
# referance answer "[https://stackoverflow.com/questions/3030487/is-there-a-way-to-get-the-xpath-in-google-chrome][1]"
a = browser.find_element_by_xpath("xpath to username text box") # find the xpath for username text box and replace inside the quotes
a.send_keys(username) # pass your username
# password send
b = browser.find_element_by_xpath("xpath to password text box") # find the xpath for password text box and replace inside the quotes
b.send_keys(password) # pass your password
# submit button clicked
browser.find_element_by_xpath("xpath to submit button").click() # find the xpath for submit or login button and replace inside the quotes
time.sleep(2) # wait for login to complete
print('Login Successful') # if there is no error you will see "Login Successful" message
# Navigate to the menu or any section using it's xpath and you can click using click() function
browser.find_element_by_xpath("x-path of the section/menu").click()
time.sleep(1)
# download file
browser.find_element_by_xpath("xpath of the download file button").click()
time.sleep(1)
# close browser window after successful completion of the process.
browser.close()
This way you can automate the login and the downloading process.
Mail automation
For Mail automation use smtplib module, explore this documentation "https://docs.python.org/3/library/smtplib.html"
Process automation (Scheduling)
To automate the whole process on an everyday basis create a cron job for both tasks. Please refer python-crontab module. Documentation: https://pypi.org/project/python-crontab/enter link description here
By using selenium, smtplib, and python-crontab you can automate your complete process with minimal or no manual intervention.

How to use existing login token for telegram web using selenium webdriver

I'm trying to read telegram messages from https://web.telegram.org with selenium.
When i open https://web.telegram.org in firefox i'm already logged in, but when opening the same page from selenium webdriver(firefox) i get the login page.
I saw that telegram web is not using cookies for the auth but rather saves values in local storage. I can access the local storage with selenium and have keys there such as: "dc2_auth_key", "dc2_server_salt", "dc4_auth_key", ... but I'm not sure what to do with them in order to login(and if i do need to do something with them then why? its the same browser why wont it work the same when opening without selenium?)
To reproduce:
open firefox and login to https://web.telegram.org then run this code:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://web.telegram.org")
# my code is here but is irrelevant since im at the login page.
driver.close()

When you open https://web.telegram.org manually using Firefox, the Default Firefox Profile is used. As you login and browse through the website, the websites stores Authentication Cookies within your system. As the cookies gets stored within the local storage of the Default Firefox Profile even on reopening the browsers you are automatically authenticated.
But when GeckoDriver initiates a new web browsing session for your tests everytime a temporary new mozprofile is created while launching Firefox which is evident from the following log:
mozrunner::runner INFO Running command: "C:\\Program Files\\Mozilla Firefox\\firefox.exe" "-marionette" "-profile" "C:\\Users\\ATECHM~1\\AppData\\Local\\Temp\\rust_mozprofile.fDJt0BIqNu0n"
You can find a detailed discussion in Is it Firefox or Geckodriver, which creates “rust_mozprofile” directory
Once the Test Execution completes and quit() is invoked the temporary mozprofile is deleted in the following process:
webdriver::server DEBUG -> DELETE /session/f84dbafc-4166-4a08-afd3-79b98bad1470
geckodriver::marionette TRACE -> 37:[0,3,"quit",{"flags":["eForceQuit"]}]
Marionette TRACE 0 -> [0,3,"quit",{"flags":["eForceQuit"]}]
Marionette DEBUG New connections will no longer be accepted
Marionette TRACE 0 <- [1,3,null,{"cause":"shutdown"}]
geckodriver::marionette TRACE <- [1,3,null,{"cause":"shutdown"}]
webdriver::server DEBUG Deleting session
geckodriver::marionette DEBUG Stopping browser process
So, when you open the same page using Selenium, GeckoDriver and Firefox the cookies which were stored within the local storage of the Default Firefox Profile aren't accessible and hence you are redirected to the Login Page.
To store and use the cookies within the local storage to get authenticated automatically you need to create and use a Custom Firefox Profile.
Here you can find a relevant discussion on webdriver.FirefoxProfile(): Is it possible to use a profile without making a copy of it?

You can auth using your current data from local storage.
Example:
driver.get(TELEGRAM_WEB_URL);
LocalStorage localStorage = ((ChromeDriver) DRIVER).getLocalStorage();
localStorage.clear();
localStorage.setItem("dc2_auth_key","<YOUR AUTH KEY>");
localStorage.setItem("user_auth","<YOUR USER INFO>");
driver.get(TELEGRAM_WEB_URL);

Chrome Authentication Alert webdriver

I have an application that generates a new browser tab when I click on a link. In the new tab I get the Chrome authentication alert. I've tried using the below code:
wd.switch_to_window(wd.window_handles[1])
a = switch_to_alert()
a.send_keys('username' + Keys.TAB + 'password')
but I get the no alert error msg
I have also tried using the ActionChain to send the keys but the alert does not receive the send_keys
Are there any alternatives?
fyi I cannot use https://username#password format as this tab is generated when I click on a link
Traceback

I suppose you mean that a link opens a new tab which has username and password fields which you input them with ‘username’ and ‘password’ respectively?
If that is the case can you on the newly opened tab first search for the elements (username and password) using an explicit wait,and only then enter values.
I had a similar issue where my web driver was stuck on the earlier page and tried inputting values into fields which didn’t exist there. Try typing your active tab’s title to confirm that’s not the case.

How do I use Selenium to login to sites that require username and password?

I am developing an automated test scripts using selenium api with python. But when i run the script from selenium rc its goes to the login page. How will I be able to put in my username and password on that page as it does not contain any sessions or cookies?

Here's the suggestion that was made to me by Adam Goucher on the Selenium IRC channel ( irc://irc.freenode.net/selenium ):
Take advantage of the situation of a completely clean browser cache/cookie history. Code your test to do the login.
However, instead of hard-coding into your code a username and password, keep a deliberately non-source-code-revisioned configuration file with the username and password locally on the client with the username and password stored there, and have your Selenium client code retrieve the username and password from there when the Selenium code executes.

Thats because selenium server starts new profile for browser everytime, so your saved cookies and bookmarks do not exist on this profile.
First create a profile, for firefox it is given here
then bundle this profile to your selenium server like this
SeleniumServer server = new SeleniumServer();
RemoteControlConfiguration rcc = new RemoteControlConfiguration();
//rcc.setPort(4444);
File newFirefoxProfileTemplate = new File(ReadConFile.readcoFile("fiefoxProfilePath"));
rcc.setFirefoxProfileTemplate(newFirefoxProfileTemplate);
server = new SeleniumServer(rcc);
server.start();
DefaultSelenium selenium = new DefaultSelenium("localhost", 4444, "*chrome",ReadConFile.readcoFile("serverName"));
to know your firefoxTemplate click here