The project I am working on finds data for cost of flights per distance.
To get the flight data I use google flights, and selenium to iterate through various results. To find the distance between origin and destination, I need to obtain the latitude and longitude of each flight result.
The links I use to scrape data are in this format: https://www.google.com/travel/explore?q=Flights%20to%20SouthKorea%20from%20London
I need help to find a way to extract latitude and longitude coordinates from the results. For example, when searching London to South Korea, one of the results is London - Seoul.
I do not see any coordinates data in the HTML page source - So I am wondering how the path is drawn from London to Seoul in the preview window?
In the developer console, I see a page response that contains latitude and longitude data for each city(See below picture), but I am not entirely sure how to extract this page response with Selenium. Is there a way to scrape this data with Python?
Page Response
I am still quite novice in Python, so any help or guidance is really appreciated.
I have tried finding the coordinates with just city names and country/region, but end up facing issues when cities have the same name. For this reason, I thought it would be better to extract this data from the source.
I also looked into using selenium for getting the response headers, but from my reading it appears this is not possible
Related
I want to collect contact information from all county governments. I do not have a list of their websites. I want to do three things with Python: 1) create a list of county government websites, 2) extract names, email addresses, and phone numbers of government officials, and 3) convert URLs and all the contact information into an excel sheet or csv.
I am a beginner in Python, and any guidance would be greatly appreciated. Thanks!
For creating tables, you would use a package called pandas
for extracting info from websites, a package called beautifulsoup4 is commonly used.
For scraping a website (all data present in the world) you should
define what type of search you want to start, I mean do you want to
Search in google or a specific website for both of them you need a
request library to curl a site or query a google (like search in
the search bar) and got HTML. for parsing data, you have gotten you
can choose BEATIFULSOAP. Both of them have good documents and you
must read them don't disappoint it's easy.
Because the count of countries around the world is more than 170+
you should manage your data; for managing data I recommend using pandas
and finally, after processing data you can convert data to any type of file
pandas.to_excel, pandas.to_csv and more.
I'm trying to scrape a website with real estate publications. Each publication looks like this:
https://www.portalinmobiliario.com/venta/casa/providencia-metropolitana/5427357-francisco-bilbao-amapolas-uda#position=5&type=item&tracking_id=cedfbb41-ce47-455d-af9f-825614199c5e
I have been able to extract all the information I need except the coordinates (GIS) of the publications. The maps appear to be pasted (not linked). Does anyone know how to do this?
please help
Im using selenium/python3.
Im using Selenium & Chrome.
This is the list of publications:
https://www.portalinmobiliario.com/venta/casa/propiedades-usadas/las-condes-metropolitana
If you click any property of that list, it will take to the page were the maps are displayed. I'm using a loop to go through all of them (one at a time).
The code is a bit long, but so far i have been mostly using find_element_by_class_name and find_element_by_xpath to find and extract the information. I tried using them for the map but I dont know where to find the coordinates.
I'm working on a work-related Python project that will give me the latitude and longitude of an address using the Bing Maps API given that I feed it the address first. While it does yield the latitude and longitude of an address, it differs in many cases from the results you would get if you were to type the address in Bing Maps.
For example, if I look up "1 Microsoft Way, Redmond, Washington 98052, United States" on Bing Maps, the coordinates given towards the bottom left of the address information panel read 47.640049, -122.129797. However, using the Bing Maps API and the same address, I get 47.640568390488625,
-122.1293731033802 (retrieved through the 'coordinates' portion of JSON outputted data). They are similar, but given that a tool we use at work consistently, if not always, agrees with what Bing Maps produces instead of the API, I would prefer my own program yield the exact same coordinates.
I couldn't find any documentation that would point me in the right direction, so I was hoping you kind folks might be able to assist me. I also considered simply developing my own scraping tool to scrape the coordinates given just by Bing Maps, but I haven't figured out how to compile a URL that would take me to the coordinates.
The Bing Maps website is using a new backend geocoder which also has newer data. The Bing Maps REST services are in the process of being updated to use this new geocoder but isn't yet complete. This is the cause for the difference in results for addresses like this. This upgrade to the REST services should be complete in a couple of months.
Looks to me your values are different by a factor of a few feet. Dunno how accurate you need to be but maybe round your lat long to an acceptable range.
I need to get files from the European Space Agency (ESA) website which allows to choose latitude, longitude and years and once those gaps are filled in you can click on "download" and get the specific file you asked for.
I need to retrieve data from a range of coordinates and a set of years would like to do it on python. The website's code uses "id" and not "name" to identify the variables (lat, lon, years)
link: http://www.esa-sst-cci.org/PUG/map.htm
Thank you
You can do this using Selenium WebDriver. It is can make direct call to the browsers using browser's native support for automation.
Click here
Introducing Selenium webdriver api by example
I am using Selenium (python, Firefox) to test the website behavior. On a certain point it gets to the web-page where there's a Google map with one object on it. I need to get its coordinates (longitude and latitude of this object on a map).
Do you have any idea how it could be done? I can't modify the code on the website.