I am trying to download multiple csv files from the below url and hoping to use selenium or any other method for it. The url requires filling up of a form that includes selecting options from multiple dropdowns. Then, an 'image' button needs to be clicked for the download link to appear.
If I run selenium chrome driver from python and click on the button, nothing appears. I am also unable to figure out the url of the csv files so they could be downloaded using 'requests' or 'urllib'.
Here's the url I need to download from:
https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm
Here's my code so far:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
url = 'https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm'
driver.get(url)
instr_type = Select(driver.find_element_by_id('instrumentType'))
symbol = Select(driver.find_element_by_id('symbol'))
opt_type = Select(driver.find_element_by_id('optionType'))
date_range = Select(driver.find_element_by_id('dateRange'))
button = driver.find_element_by_xpath("//input[#src='/common/images/btn-get-data.gif' and #type='image']")
instr_type.select_by_visible_text('Index Options')
symbol.select_by_visible_text('NIFTY 50')
opt_type.select_by_visible_text('CE')
date_range.select_by_visible_text('90 Days')
button.click()
And this is what happens in the selenium driver -
Any thoughts on how to download the csv files from above link? Doesn't necessarily have to be using selenium.
I don't know how to resolve problem with Selenium but I now how to get it with requests and BeautifulSoup
This page sends your options from form to this page as values directly in this URL (not as POST)
https://www1.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp
and server sends back HTML with table and also all data in <div id="csvContentDiv">.
This tag has all data as text already formatted to save in csv - it need only to replace : with \n
EDIT:
Sometimes server gives me Status 405 Method Not Allowed so I added requests.Session() to get cookies and maybe it will work better.
import requests
from bs4 import BeautifulSoup
session = requests.Session()
session.headers.update({'User-Agent': 'Mozilla/5.0'})
# --- main page ---
url = 'https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm'
r = session.get(url)
#print(r.status_code)
# --- table ---
url = 'https://www1.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp'
payload = {
'instrumentType': 'OPTIDX',
'symbol': 'NIFTY',
'expiryDate': 'select',
'optionType': 'CE',
'strikePrice': '',
'dateRange': '3month',
'fromDate': '',
'toDate': '',
'segmentLink': '9',
'symbolCount':'',
}
r = session.get(url, params=payload)
#print(r.text)
soup = BeautifulSoup(r.text, 'html.parser')
data = soup.find('div', {'id': 'csvContentDiv'})
#print(data.text)
data = data.text.replace(':', '\n')
with open('output.csv', 'w') as fh:
fh.write(data)
print(data)
Server doesn't send data if you don't use real User-Agent - at least short 'Mozilla/5.0'
I found this url using DevTools in Firefox/Chrome, tab Network. And later I get response from this url and manually check what I get and I found csv data. But I expected that I would have to scrape data from HTML table in this response.
Related
This didnt work for me. I am scraping the website slider.kz. But when i scrape the website it pulls the html of the main page instead of the specified url.
My code
import fake_useragent
from bs4 import BeautifulSoup
from requests import get
headers = {'user-agent': fake_useragent.UserAgent().random}
soup = BeautifulSoup(get('https://slider.kz/#unravel', headers=headers).text, 'html.parser')
for i in soup.find_all('a'):
print(i.get('href'))
The actual output returned:
https://twitter.com/x_slider
https://paypal.me/xslider
/
What it should return extracted using selenium :
Selenium code used to extract the desired output
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://slider.kz/#unravel')
for i in driver.find_elements(By.TAG_NAME, 'a'):
if i.text != '':
print(i.get_attribute('href'))
output
https://slider.kz/download/371745468_456554892/238/cs3-3v4/s/v1/acmp/cOE3Il1BHU9ESEFr9Hz4pL7WfYMASX1jY3IxqYGfVbDA7njmXutAbM3kHNfpwOzSBQWa2IZQ7huupcnHjAdwPgptzXqreGX0tmqzvcVCEsQn7_VVdO2neH6DlOB61uNtIBCcnHAMlApXwBfbMccaKZDif1_ZPf1e-4rkVEyHP9VQBiCGZQ/TK%20from%20Ling%20tosite%20sigure%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499200_456477539/242/cs3-2v4/s/v1/acmp/68YuXV09VzsZx8eNd_eKHNOjhjlx7Ae93T9h6eA4wjSJruGLjWdJqvGRkKfvEla9GYPHVopebj2BMYIsTf9R2a5-ztCHImKELMaUZQBuM1_OtyVbt2_AxQqxfs9ziB-1F1_vSJ2wJQO8EfhsuQi8Ouz-qH5nyqJPMAaPhuKuqeJ7XDdIew/Anata%20No%20Senin%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001620886_66620886/237/cs3-1v4/s/v1/acmp/9F7u-H32JUQGkAjlBPuZTbwaM2ZkjLchrhJxmqDZjIODWGIF6ClRnA49X4_rMHf5E8b9SPQUJqKtEwS_4cq1EkwDf97msqqps6gnKzAwJRNU7lhLkaJ6hqd0yZg27uBAvK6wMpizrt3HpGUIuURFmYDMWX7ggp802aWW51gkn_rQOAE0Gg/FalKKonE%20-%20unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499141_456425057/270/cs3-1v4/s/v1/acmp/sD64r4KNC9Ye4CfOMggOFmSA-ric7pUSKLaN_bfGIArMK7MSwjhu4lHpICSxWKCDniNKGTOOiGcg_6b6VUfLRuvSzn-Cakp4YU2BFw-5DJ2KAlFf4Duz6txtUAen3mslUB-vOSV8mT48PYf4VuoxFkltIb-kYqzkXgvsr56GGrXFzYo54w/Dima%20Lancaster%20-%20unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz/download/474499133_456468907/240/psv4/s/v1/amp/I4gkKu5n8-ePABgx_5blRV5izpZRPdm6o3_5C619b-f0tDciBsojh2EkLvtQ3xref_TYPVL7vBktamf1mdxDmxi3xZz1XC_HKAwsjUqU03dJpvytOBXsiCZ6kgP2FQ/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001476389_94476389/229/cs3-6v4/s/v1/acmp/A6FBXBqjCCmhlytjFG4rAKFNfD78JTsqzgNAZToGubFqSZ_t1OQ_4gXhqO9qxFs3nPX2V6u745hB34NMrpGSvAEEKhfV0u5jYjHKeAdqBTbp05hvkdg2aJ6lJMFO3gjA9DZjanfjDL1fjhDC02e50HYr-K-ILhQkjdrixmzi8Ed5PQJfXA/Nika%20Lenina%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001355458_97355458/244/cs3-1v4/s/v1/acmp/ZRd1eY7d7KVAi23V-pkv91B0rBxyH6TBawtgTp0u_agc_1WbMS1061fhyFRFye9GHrUcaKX2KI1HIEzV0SXmJbSgy3q86HHZq08hTiwHdE27YVaxwn2HqW0r_AmOLtPMt6EcSNjcRyKUc-8sPnJ-PYKIPGeHoF67WUBsDbBteD1LjaG8fQ/TK%20from%20Ling%20tosite%20sigure%20-%20unravel.mp3?extra=null
https://slider.kz/download/474499285_456610507/236/cs3-5v4/s/v1/acmp/yQ75TttqYQXX7TBSoFeRrACN7JTw4pDVbn7gQ0uYHgKyUtvarViLiz47siDjWti8wr-Udt9pYektb5tp-VY68B-nFSq8SQ1Amqo3YCROcOFMhfA-xxT__aZOT2eLjyC_8KqHeLGi6Dzc-Jo52P-30wiWCh09ScW4t8rmQGjspFzDuWMkqQ/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499192_456485073/91/cs3-5v4/s/v1/acmp/CnNkh81L0geRARmPlCx6HDtwarMIshv51_2gNlOHnFHKfJ3sTWbOXjzT7-SBqMmIMIusJ8lRN9tsjx4djCww8g3zd5n-x8SxDVVcMxhXKx-XqtOdT1q88mABUQhKY4ZcLos_XBYe7lx7C9CzcBabYFpBYZRZ-9x9M-HX52DfvajtnggW5w/Yendorami%20-%20Unravel%20(Opening%201)%20%5BFrom%20%22Tokyo%20Ghoul%22%5D.mp3?extra=null
https://slider.kz/download/-2001941575_98941575/185/cs3-3v4/s/v1/acmp/fmdPR0KKIvzX-evg6umhtRemEihc16Hc0s0W6fanPFxDNR8vTeCaTGdId8L20cp4r43dNQO3XYNbdp4ftt0mwIxb89SFMmcn5bBMZCymCay9cegxcXx4dDhc2329j0Evv_KSB3wpIzA5IR9cBf34DyOdDkd6L1b_ygIpDGF5E1_yp4acPA/Collosia%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/474499243_456360717/225/cs3-4v4/s/v1/acmp/i3ZW2FQYrSPIkVmjzhKxasRpNU2sHQA24dwNyYtNb7SpQTg9bTqGlKbpeknyi2-z74quCFgDKR6Vox0JFmyJJVHQmQkGrPxTgnngADz-Iaxa4q3MVUAJqzTZJgu32Ur-FRa3n-lcN3wy/Trinthepianist%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BPiano%20Version%5D.mp3?extra=null
https://slider.kz/download/-2001187532_103187532/144/cs3-5v4/s/v1/acmp/NuVJyRij62WtsZrXRq60T7I1G8mfLu-Q01ITWVXxnn9DvIa9hiRGNUfMX2HF7qJMNv6nLC8GzobY8nmh71W9ulIpnXI44tGkyBKcy3lyAq1rTpBjLXssQiC94zA07P238siyirG99-EQsSAFL29rMcG8UYN9hXqC5g9-Xc4hyhT5vuNsHA/KinovariSquad%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499138_456497179/330/cs3-1v4/s/v1/acmp/OClerb_S3_yMpf46bD6zx0ObnCHcC6OQTPnK5Pry3qn8XVTOSLd_xRzCzUodFJoaLYbrjnKNHktd_J6PZu2m1hkZM3N--UNOqZkPIMubcOCJfcBBloN5BoCve2WvgftZ3yWubFzzWHiVpLnosbeogAzf_bMpblrewEeJ0-xC7toQeGTJ6Q/SLSMusic%20-%20unravel%20-%20Slow%20Version%20(From%3A%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001343575_62343575/235/cs3-4v4/s/v1/acmp/O6pRAZYWRIaUHjj-3d-I7wDamPE2M1MziJsmpF1XKCPGPbOKR3UV_A5vqtpbBqPSPochs34hpYKTTsHS5eAL_FiKJwwKvbUFqD3exRAAbDouMGxfW7YEpmMkN8NcpVz5QmH0lEJypGWO84b9cRnPSB8ZAkWNyKpGue26FP9Qc-jFLAj6Kw/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499226_456421440/235/cs3-4v4/s/v1/acmp/ErptzsuuaoawWhx_finD4Pb5GRbG05r7PrMIQy5LYmfw9QfJqjPBRGK-vRTrZnbsfSKoCA8K0odPCNvb_3rC3Mv-97bmbyv188pzC2PIKET0DntUvYYE__cD-KCVTnTyUpx3M5fe5hNdYjsb9UkYVngGlSBMNrA4OQhrp3-15Fybbvrfsw/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499206_456447563/243/cs3-6v4/s/v1/acmp/Ll3SqtO1lm7C2W7j7T6-bCEZ03fOS_MAFwvXUqzizClbi3AJS4LYDddyPM_iI531Tv-t1hUYKQFeJCUsF9MCM_ox93UUSaP_M9Zp0uVdB4PSpBXcu6xgqoKCDkbQ69CcnGpARuXxeZ5XPo6hJox-8xvMuZnfoT_b76BasOFB7rcYjFPDsA/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel(%E3%82%AA%E3%83%AA%E3%82%B8%E3%83%8A%E3%83%AB%E3%82%A2%E3%83%BC%E3%83%86%E3%82%A3%E3%82%B9%E3%83%88%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E3%83%A1%E3%83%AD%E3%83%87%E3%82%A3%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz/download/474499222_456568196/98/psv4/s/v1/amp2/G5Qoz-PWlD8w86QFGNvVLmXfps93Kv7CVsbbVFpo4e5LjDQRJ_yuY4Ha3Nrp10xVXf5hur40alIvROP88ncZTLhPICCjkBn3mM7QGpaFgBQf3SEib-JCMELaZoMg36gpsc6vdsRKzSqjecjnhDAvjkixFTc05wTyL3b7/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Winter).mp3?extra=null
https://slider.kz/download/474499266_456496174/238/cs3-2v4/s/v1/acmp/ZSnEUsWBGhnXj5XDpV5_RYtliMXOCl747j9MfPXEJut-jDHGl9lI3z-Wyrmo1fnrb7QYMdmCyCFGChtLb_Jlnhj47w9llLDSfOeT-ejt6HrEFCVeG9pSJfb1x2Orv9ls1AWiprtX9TBylxyJYkTTMCT8lXPxyT_1Lhh63I_Vclx__QWHAg/The%20Unknown%20Songbird%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/-2001091207_65091207/243/cs3-4v4/s/v1/acmp/KQtj35XBV_qATTaJDm3ULd5piLxC8bhqrobCvIFfHM35M5YDIBlSZQyfTvcmGktv5upmL40wFiQuX65il7CSkTyvGgCubXJXch30BTk_8VgSnOQitmFNebJzW_ADxbaBwFpJsZRZ1La5UKahLdafzttj-qiWjk7B9uuiI2PejUViDS7oGQ/Bibi%20Gamino%20-%20Unravel%20(Opening%20de%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001815387_86815387/189/cs3-5v4/s/v1/acmp/ao_2H9LXIsXS3wGH47_DjS8MTmTGiH8wBvOjh7ZTyXXT6gGCzwqO9j3fW940H4MSUnTlarmjLDf4kFN9izLPboP6opybIVo1zv0efcOASrk03xIjwIFbsZuB7Ub1vpl_U9jevxKne9xT7-8ecNhrm8_MrrcqEt2yUw6ApNIIDiFIjnPm9Q/Baur%20Karbon%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001343001_91343001/91/cs9-18v4/s/v1/acmp/zwfryFVsrFggm-hEVOd6RiXeyEUooW9tZMLBrntNjiPJOxatNavKMSQ6tdcBDb5DfN-5oz89p6eXNd61Td00hMgQXQ3pfU_uRhQXPtSNRXH26a_vH5x1z6VVndmwxWikty-kmyuctXkSLygSr-__9qnxcZYaNplAvE79Hl5maQC14zIq3A/Ren%20Avel%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/-2001459950_102459950/222/cs3-5v4/s/v1/acmp/dXsMP_kZ98lEdBVsmwcSr0zvYGcr5Spyn8RNNVYxfMDehWA3Jbcq7BpGMLoaKcgCMpQ4PpHszAAC3bkW4wNbKiFD6LumIhx0_T2C2xekVJsfR1xb8cVtOgwLUMFnffOWz3GULiHXBKoPS2xjpWf1TmkQXdDhcpmKqokCzhx0yvKBagDO0Q/Jonathan%20Young%20-%20Unravel%20(Full%20Version).mp3?extra=null
https://slider.kz/download/474499249_456500220/95/cs9-5v4/s/v1/acmp/kI7HhI6sOV-jdnMnoifzOJwbZwidKai2LQ6nQOuVeZ73W2eyOcQUDIDLEuDWL6qhNZM4ATnhITDeeI1W34s3kaY03Bf9CGzRm2RTJHtIbF7U1ZHL5GJ_lKj8DR27Ulz_4whswvN7p4sUAuL8LHEyYCFjLrgQVlLTb_xfVL9j7q5_4ndF5w/Tony%20Gu%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BOrchestral%20Arrangement%5D.mp3?extra=null
https://slider.kz/download/-2001472007_90472007/95/cs9-2v4/s/v1/acmp/D25FbaAzhvubfYuHGk82DiPeG6gHBuhK6YwgRt3UV2gnf-s7ynvbhcNomwhgm7kqj6RZ5cqVRgRhzUIVCLvLPjnuChQiPRefKOp8IH35ZjCBLRB5-CWFnNKMc2Z3Z_jQzeyxzikPI9PI5u9fErheiCWVb0HnR5k8FSg_guhCQaOBeYKgbA/Radio%20Control%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499153_456452837/238/cs3-6v4/s/v1/acmp/QOuii-Czus3bM2CyRJILUYMWsCNb-6TEgDNKyBO_qJYdxeoLM5YEqyW03di_3mSuFpbTNqCD8cEN8ijmtjMRESRZCKya9Wj89Hyo8A6s7oQFURgmc0OAQw1ifU6FNPgNuLt0cM8UuVu5ONMZcXLxQB_X-vyVga9enPIDvzadwIPzMLrkKA/Kevin%20Remisch%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499122_456591314/97/cs3-7v4/s/v1/acmp/lOsM3wtU0cPI-POYId2zCQKMuFocZYxke-zM9kGXjLh_HWnKmpt9FT_BoUqdG_QMSQyXfnUdjzt11bJW4Z6hkc8izzQ_xo7UA70AhMVu2Ik-d5INvNV2W7xAbavVROqzFEPxMEmmYpazITlYQEy07UapD3KiFWUY1WRDbr_-1v30o018GA/Lil%20Bubb1egum%20-%20unravel%20but%20is%20it%20okay%20if%20it's%20lofi%3F.mp3?extra=null
https://slider.kz/download/-2001077985_82077985/238/psv4/s/v1/amp2/VllxAjiDhUD2kGGN_1QYYlEi6ljH-VawyGBR5t7gRwmBd4bnyoy7Q9TP42q83gu6qs0Q3xjHT7HEgi-HkPQAs3w417-ESpSQGMQEySHxg8O43I_0gt0dJ6edsjnIsWzRIrOtSJwBSeqtoUKitAfdgVRjP5-6egt8gLqr/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel%20(%E5%8E%9F%E6%9B%B2%E6%AD%8C%E6%89%8B%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz/download/474499138_456433311/238/cs3-6v4/s/v1/acmp/VNdbMoXjNPrw1jhq0upwNnZfbolEFriZNKeUEHSf1ncOJUmACaSbyYEZiu1N8JghIVspFPzHWO3STGmE0lGuFvgo0bF6Y74ykyvwwSiDgcRqMbUBbn2ySHSA5RSLN8nCinRGvDvJ-5YSbYmvgZbwaNVWVFVJ0R7D4gwA5RPvt7BQj7eaeQ/PelleK%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/-2001778472_74778472/238/cs9-2v4/s/v1/acmp/z1oe_35fnQwSwR-yAjBnQ4MlCd2PNIEnRSLz_ZCKF_5PldV40_lerssvNhZazYLx7YpdpVjqm_HNA8PIuAqrlvKioKNyrOu3GL048jLLCz_xJL38tDuU22qmn8mhf5YJ7_XzQygG4tmjHFh88hASl-wYpTowxw4kmOAcS0perZf4inwKbw/senakisaragi%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499220_456599167/246/psv4/s/v1/amp2/ZyCoku2jGfwD1BaUlyq7-q6TDH-IwTCPHDVoyJSz9TRj4hLvzyeFs3j5um5duLh8mru1sc22DqkxA7Ajara2-Y9GKl7PomX_F88WidLCB1MARvkw2iYrizN-q4Et4QsAWJm2VaSamp1vRDksmw5SUgN4a74Qh58sRx2Y/Matteo%20Leonetti%20-%20Unravel%20(Japanese%20Version)%20%5BTokyo%20Ghoul%5D.mp3?extra=null
https://slider.kz/download/-2001804556_66804556/238/cs3-5v4/s/v1/acmp/gIyB4Vft-_97FyJAE9_0hQlav32398RrusRMOaXgEkSsoBNW93224d76-9tGbFvAT07TSJEwetYGewbka3fjyJfSui8-paQTAJMVI5lMtwgMuTU9e6fB_cRtlIsXuN6vZhQORVl47To-JGxYgfN1-zRNimSOEBj4Ve_p121ne1UwhTnDLQ/V0RA%20-%20Unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz/download/474499291_456568508/102/cs9-6v4/s/v1/acmp/fKvZmSOyvLXOxRXgOatLErO-EzVCD2cn-tB1Dys0NFpugNtd7A4hZlhBPUZMXAKZIevqS49l2ZAscFh-k3RRHX02kgrbPM6H6AhcAZ_jbvNstSTtxywJvz68KeUWJ5y0Bp8ll3fL7g-rNSYGQEzIxoTjQ6LXqfB1eolI4ZUsqC8vUfoRcg/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
https://slider.kz/download/-2001227120_98227120/186/psv4/s/v1/amp2/cLi6IzWrhQhwJVpx-nbHAVgRnIxbVTGQQZfcqBf1JyjYIYi55BhiY6bc_fFMdWxKV5lMiYXFcAbUTzrketBr5NXGs3a4B682hU58z4nXP-E3AET0RRFi2-zBVQkK8zeHEOfBGk4-RH8HbJvY4f_oJHfeSSfhVPyxOKdg/Olga%20Scheps%20-%20Start%20Anew%20(from%20%22Unravel%20Two%22).mp3?extra=null
https://slider.kz/download/474499232_456549884/90/psv4/s/v1/amp2/G_D-kC0LEQYYW7ekPdSMg06ozpZssPpm4_4ke1P8RhdHN2a8EH0t27UjRhyBw98-veAur6NiLRcicS9vp4nzOPQIsqaPr_-v-x9h8m2L3yIzpDcVKTftL2Th9rB_H4g4tHTUl8orsWQU6npEA7sTWysshIwpFs_nqWQq/Mary%20Antoine%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001667855_68667855/186/psv4/s/v1/amp2/xznk6i9L8LRfJar58xq3PjlC8dSayqHAQnXP8JIbQTSUFF6dtmf4AG1_Sr7dR-6gYxW05dzwebqzJJU-KkNJvqu6ieBiK_trxotPhLkBA3mmBcskTm8z1q6VnpE3YPOYyiJIOFl_NLvxbdT7GqlNW1UYnKScRVumta8m/Track%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499146_456450407/186/psv4/s/v1/amp2/wEP-qGrwzxqR21c80CCBheBmiu2fIZVI4VsGd7CuyGxTkVI1boxILGBdheTs6tX0N6XCuLO1j4bAs5B8XGAmAEdD9ijXYGTW5cOYDQ7psn31PqmHWKubTdiefBdP9ztaYS6szPgmhtQ9FN9OoKKOH3-WWs3qW_IvYAFM/Track%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499318_456592131/96/psv4/s/v1/amp2/1Ngs7lG1WBh6TJ1IhzauBhSNBeazCty_k25JatXLvLqpadvgBgoQ0lY9ISeBhfts6PV2hs5QtYk9EZFXczQq7GVshKDrg8ovPv0OnPTolvNdOhhJjLiKPoF2dUewl09LW4yvJmX5ogCX7q_Pq-F3bWeLpa20NankBa_c/Vangakuz%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499173_456432829/137/cs3-1v4/s/v1/acmp/rT3ABEaA8kmBYfjWSUgBN1i_wGr6c7BO5YQXfOdt474aADbZ0Tjg4OWB08dgPLHIcx6o3-rRRELc59NKPIhWhfVQq8jatvyFzxR6k6bo-wp5dJFVptg21nj7NTtWPOMcgvjXxblmVfZrXtZKGKCfFoqb-sO2sDKOM3prAMESLmk_HQieCg/Kenzie%20Smith%20Piano%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001400485_76400485/187/psv4/s/v1/amp2/wQrrOm95FEb1EpWPN05KTxkNgjEfDh_faLEMRAA1Qk4F4cCJSO_9sd8XjDlc714dR_OK9JyY_9AQCe6moPsLeNnM6iCbXzFZrdWObICKCxggwhxuCtyaM4liHWk5TgNcFC1GLQp8Lu7Hh8v2yiC062n2nfsvZHjenrho/Simpsonill%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001521828_57521828/551/cs3-6v4/s/v1/acmp/dZXnCT1fUfJ6PdA2mzGe8yHpA9s-DY_9UYw3SUYt3z_QoxPkXt6m4epU3ilm_7hieMGGJIe-HQeq6UYmj1QZYvJPezdSZyd6wBiy288Zdv2zwMEqQsWrKziLnBIDpenTzJfh8Lht97PQ4hqkQ7mOGO8ucyZm1evH5kQFymrzeFQcstVf5A/Alveol%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499237_456436594/88/psv4/s/v1/amp2/8Bo41ZWBvHso221zqJhGjyABsxjNh8o27U_XLhVWL93AEuMhcNxyhU2Ur43TuL48o6nFZQDF5HpQj43ra3EM0ddi_ptqUCJRgVr1vF5ADTqFXYlzqdIO_Jg64Vku2kEfVPwd5KYOHoJD_VCsRTFN83sW5epkBJm1c798/MattxAJ%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001915683_101915683/90/psv4/s/v1/amp2/QaH-avVk5DSEc-4Xw50z0JX3TZLUGZ0nk4PblwUkmscM8QYZh2u6t8rFGB7D_GzbyTwxltM4Rp0kE1x0WfCO_QHO5IHZe047S93z7D2a_-DEnr8v_bwDtLv_10NQRabdX77HE8h9iWlSs_doa76LQMm3Gm6fYtCutX29/Music%20Legends%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499272_456394915/236/cs3-4v4/s/v1/acmp/WDQwU05V9M89alMuCTi786gxWCQDoeFFwYlNzojOyQ5G5oeCmgknrMPfQGlX1dE1nn-JHFfaJOOhgXQJWptsR070hN8OufTiDgs4dzg43TzgDhi_Cn6mXxnfAq3qK5xDWSY4Kamlydi4hVH6xcC5uRrpy70lK5xEce7eq01yornEDvVkcA/Shiroku%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001529426_101529426/255/cs3-2v4/s/v1/acmp/yz7am2kIPxn76UUaQEQFAeOfeTQciBwOu8g151LzAMMKMQ5NRg6nI92UZBjPePHnH9rV7svs2udN-fiGIpg2s8QqIbP9lWCyG-J1qUB9CywlXwu8tfg6CzSTN2GgbRp8iQM-QKlGrRgLxi56Up3r0U7oWubLYL0hEOKr3ysemG_9tB_Q8w/HalcyonMusic%20-%20Unravel%20(from%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499176_456433059/241/psv4/s/v1/amp2/EZVr_Le7Wnc33wId8JajbbOo03wZBaoT8f8_ZyHVTiL5h8hs8HtGSVdm3BLNaW_AH5znzjmROfyAYYPoYnLQ-pY789RfVtTwX034IVdl_tM097Dube-bzIGaa_7COKBlo3Dp_1lHWfn1EjrgHEI3YhYPhnVyxOjkRsrd/I%20am%20Justice%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001621142_58621142/68/psv4/s/v1/amp2/Oc2oMY7m2oW3hfpZDQIW24g3lWZIUDD1idJn8Wn5ZN5LJy29qZTBNwcHYCcxvtLPz5YsK4hWSvi71bW03O1eKw4PhvRXcqWreiPMJI9BJsHU6vwjXk1OieqMk3zC-stJLo5ZRIMxUpuefWK8HQlv7HfCAqrba3PI1_yG/Gianni%20Tallone%20Della%20Cavallotta%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001052300_100052300/89/cs9-11v4/s/v1/acmp/jENDq0_xjfWlA1Fau3Cd16w9dQT3Won_n8YIMUI3daFMPE4P3PEuYPxpnn37TYSDXtZsXjyDSPLFO2HbnzGi3yDkvfePGfIrY2lK4y19tTEY9Y6aRUZIbAN5eS-IWG91t19G8soWuPs8B09mV82_4lj6lW3900E11hWClLA-Xl7jCoN3bQ/Fonzi%20M%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499157_456624310/212/psv4/s/v1/amp2/WpHC4MCzE7o7pBv5lzq2K9K3pf4fZPUgW_GycJUP-92L5fd1yNx9fOLzJol5M9F5Mkwv8BApcjZkjLw4nC75G1rX7bnR-oZNOBP1pSw0EgJZi1o_a2soIKK3YLTMmcHdcdxXGNTJ3IG0YasN9w9n_8pzlc8mYoQYkOf1/Love%2C%20Lies%20and%20Fiction%20-%20Unravel.mp3?extra=null
https://slider.kz/download/371745435_456449924/271/cs3-2v4/s/v1/acmp/2vGBvFG1ejsCoSv88vOr1JwaXPjPCw6aFRXBB_8fqa1j_LxgWXH7bzy_9JbcymQd5ACAr1bYoLnwBwfthYgqYHMWaL2DGTCu5Lp67BHZXWHRyGFeD3iKM5OeKLh4642GtJSGxVkgApp_9cRCvvIj-sRUAI1a0jgd_zjfAafM8eE8pf8WTA/The%20Blue%20Notes%20-%20Unravel%20from%20%22tokyo%20Ghoul%22.mp3?extra=null
https://slider.kz/download/-2001260329_90260329/94/cs9-15v4/s/v1/acmp/9QTEytbysHCNmBriy3cvC2OOMqLUBoFMaUgslyo1b_QU4KGSKxWTJ-1x9bmqlxOIaCKJKTrsStWyE01pTlyCUU-tImpXxgFKyqcdpwQOvNAtLT_jiCrxmVpKGGryMeOgDp7xqcsbwWrDTqUYh685cY_Gc4avJ7va_QDg935M6qq_TxcxIg/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
https://slider.kz/#Contaminated
https://slider.kz/#Excarnate
https://slider.kz/#Extremity
https://slider.kz/#Serpent%20Of%20Gnosis
https://slider.kz/#Petrification
https://slider.kz/#Skullsmasher
https://slider.kz/#Hissing
https://slider.kz/#Mortuous
https://slider.kz/#Skelethal
https://slider.kz/#stabbed
https://slider.kz/#Winds%20of%20Leng
https://slider.kz/#Usurpress
https://slider.kz/#Encoffinized
https://slider.kz/#Glacial%20Tomb
https://slider.kz/#Overlord
https://slider.kz/#RESIN%20TOMB
https://slider.kz/#Descent
https://slider.kz/#Hurricane%20Death
https://slider.kz/#Gutter%20Instinct
https://slider.kz/#Crypt%20Crawler
https://slider.kz/#This%20Game
https://slider.kz/#LIFETAKER
https://slider.kz/#Autokrator
https://slider.kz/#Vanhelgd
https://slider.kz/#ATARAXY
How do i fix this.I tried randomizing my user-agent but the same result.The website doesnt have any log/sign in.
Chrome version 98
Python 3.8.8
This page uses JavaScript to download data and generate links. But requests/BeautifulSoup can't run JavaScript.
You may try to find url which use JavaScript to get data and later use it with requests. Usually JavaScript sends JSON data which doesn't need BeautifulSoup.
Using DevTools in Firefox/Chrome (tab: Network, filter: XHR) you can see URL which it use to get data and try to use with requests.
It gets JSON data from http://slider.kz/vk_auth.php?q=unravel but it gets numbers, names, etc, and you would have to compare with real url to see how to use them to create real url.
At this moment server sends only messasge error so maybe too many users try to get data
import requests
response = requests.get('http://slider.kz/vk_auth.php?q=unravel')
print('text:', response.text[:100])
data = response.json()
for item in data['audios']:
print(item)
It gets also other JSON data - similar artists - from url https://slider.kz/similar/artist/unravel - and it sends data without problem so I show how code can looks like after getting data.
response = requests.get('https://slider.kz/similar/artist/unravel')
print('text:', response.text[:1000])
data = response.json()
print('key1:', data.keys())
print('key2:', data['similarartists'].keys())
print('key2:', data['similarartists']['artist'][0].keys())
for item in data['similarartists']['artist']:
print('name:', item['name'])
print('url:', item['url'])
print('image:', item['image'][0]['#text'])
print('---')
Result:
key1: dict_keys(['similarartists'])
key2: dict_keys(['artist', '#attr'])
key2: dict_keys(['name', 'mbid', 'match', 'url', 'image', 'streamable'])
name: Contaminated
url: https://www.last.fm/music/Contaminated
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
name: Excarnate
url: https://www.last.fm/music/Excarnate
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
name: Extremity
url: https://www.last.fm/music/Extremity
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
name: Serpent Of Gnosis
url: https://www.last.fm/music/Serpent+Of+Gnosis
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
If you use the browser inspector (dev tools - network tab) you can see that, every time you hit "Search" it makes a get request to this kind of url: https://slider.kz/vk_auth.php?q=unravel
The response is a JSON with the results.
So, if you want to use requests you should get this url and then extract the information from the response JSON.
Now it's working.Please just run the code.
from bs4 import BeautifulSoup
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
url = 'https://slider.kz/#unravel'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(4)
soup = BeautifulSoup(driver.page_source, 'html.parser')
#driver.close()
for link in soup.select('div.sm2-360ui + a'):
url=link.get('href')
abs_url='https://slider.kz'+ url
print(abs_url)
Output:
https://slider.kz//download/371745468_456554892/238/cs3-3v4/s/v1/acmp/QRhl3TT5g1GEHPv-bkQJXT4UPbfj2OAAOI01kg3IFGV3PQUawPjulWC4U1oquAFZgADZF4QaOxk8CrEfBzLih4EpJf2a8qTr3WYT0yJQiCNDV78tnhuZAGmi1p2I4wwdw_743Rfe6Y9lvMXcVHwnKd7NdRyUlKLz5XxjVAUTGxP17pY_qA/TK%20from%20Ling%20tosite%20sigure%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499200_456477539/242/cs3-2v4/s/v1/acmp/AsZGD44ViRPS8iviU4fbOjb7I095Guqb8LMt4EBRZEwgTYR2BFlFAI4HtLEu_JTeohwHwakRgcJ9pnUurkL98oL6co9QTzepES6zqJ_tDYp4oUN5D3E5pm-m2LZGkqeCNcLsZmA6PeuxNSkpEvbfnq7W-in6XtIpASiP9qQ1UdSb1wXIgA/Anata%20No%20Senin%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001620886_66620886/237/cs3-1v4/s/v1/acmp/gOSXhnJZc_7qVqSRt7soH679KeuB1qEa_Odx7P2YY9BwEAbmB9L2N-j8Qcm2roEWAxA-45rEGOQ1KT3r6CxnNYFZKC6t0Gg0J-wIjnC0sfDNhJrQpogT_kHrV7rhUSyFPLq6mrP3A7Tj1Dgl_KzZdjB-REw-21XuVyxlrLIgQRJiApQ1vQ/FalKKonE%20-%20unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499141_456425057/270/cs3-1v4/s/v1/acmp/9l2EX9ERaANRDGCKW8RS4-_VZR2MZYDlMRLy-cTAlKW4SP7QsodWeBObK7VaJa3vQiD7DOC6YEdOxTZHUUUksY6Xj7l1OgMyaJ1uzDLiVxtTy7NfJ98tdknxtnDhvNk3gXnoTZI5Fhs18fSDusmQkFAcDYkC8A7Czpng3gLvsvtoHQeQjQ/Dima%20Lancaster%20-%20unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz//download/474499133_456468907/240/psv4/s/v1/amp/84PrmRmfu7EBaFB3FMcMuLy9CrIgecvqzu74ZHTGdBVyhe1rQRRhSBfVA56cHNnwSPP4GNXVvQ88IZJq0exhcN8Fvn5k9rIdI-PJWu8QviAWxX4aYZqon1Sflf1Cbg/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001476389_94476389/229/cs3-6v4/s/v1/acmp/YRjjyO0TiBxN0blkh9T8e-6g6SepkCtdIMtBqNLYvHbfqOCYLBqpvwS6RmWXLeYNUKMy_FutPbQ9LLRPSHFO-TYBQwAM4AoT6Z5_vvSepusrUJu33dDVYMAIWntUc8ABhjmnLRfVPeBbHYzGnx22n2Je_U15qYFD1we3aWiaw2qV9CxbTg/Nika%20Lenina%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001355458_97355458/244/cs3-1v4/s/v1/acmp/f-WX05d925zpwPtfj0RS6oUftizqO337PogbUWwnM1M8hsXNV26bPGdtEgze2yNpsEQygL5qeip-xc5s7LBFOtY5X_4n0fQo-oWz9vkdRvD3sUl9XEPgibbUob0VFJ4itJXtm1l1D1LRHTyDcNeA3xDj2S2n1s_7Bi59L8RpZgIGWmvtcg/TK%20from%20Ling%20tosite%20sigure%20-%20unravel.mp3?extra=null
https://slider.kz//download/474499285_456610507/236/cs3-5v4/s/v1/acmp/QMrplAB509ufV1tZgQn756q7IsLtpEmITRyaCLzQBrMEFmK-Zc-IkEWdOI_eE5D4-Nz1i2Y97wu7cmlF69HxYWPqjcp-2pWLuUzg4h1YeOUtTGh-nteiMEdfnpJYfYkyP6ZwayqLFLOh7OH0Ki9n3lNYRm1A_U47zVi-lsRS2IGOCZQAOw/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499192_456485073/91/cs3-5v4/s/v1/acmp/ukuxjLP1ByDVop0pyoGvFwd4eQTFJnLH_6Dod2uT9uR9VCLHcTk9zoKqes1bAAb7bM8wDj1VTxi_ooy8I_AcNjW91hjWhb2uKMBYow8RR2WciYlyJ2ndYfqcBHwTOH_bj4fd1nPng0AFGG7phKNleCZW0tg3tlvzMjTHPKfsOI3w-Rypug/Yendorami%20-%20Unravel%20(Opening%201)%20%5BFrom%20%22Tokyo%20Ghoul%22%5D.mp3?extra=null
https://slider.kz//download/-2001941575_98941575/185/cs3-3v4/s/v1/acmp/kLoq7G6WCnnVNV4gIOaTAA0sZWVTO34BChRHQ-SLhB47bp0Fu3ogijAaADtrwsqaZlKDLkY6RwLlAf8f-hgIGM9TCJ11mwzLsnIOc9YJRkDeUYH74GT2WF0tIO7nlPGg9iv2dVFDld089JM6-XcbT849iJCegdNGQ3qSkcaFjrQ1NaJFbA/Collosia%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/474499243_456360717/225/cs3-4v4/s/v1/acmp/N0jZrOAfPu-5bJlwP0HzVU_8yG96Xddn12wE1QMmBOH_3aza1-ig-cHZHrY3dx76Ahm8ppqKO-uYXOCNAYUEK_mPy2Uy6npKbUlnRKn-YHIze3gP0IRvLHaaVMCACfEx3W-srnsSQTiN/Trinthepianist%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BPiano%20Version%5D.mp3?extra=null
https://slider.kz//download/-2001187532_103187532/144/cs3-5v4/s/v1/acmp/TYPoa8_kCxJuIGNUL1Wl_5vYYnxcl0CrUNYlv6FzMCLCgRyI5TrNjhX7xCuuRFBsSxGeTDX8rqI8acAvhNSGX0u__Hzr84NWTzMp2rtgee6NyN793lVcVF7r499W_34RTTC6exIIGcYEIhwUxDMC-BBwIQr6pYnBctsAbVJYd4ZcD-2JKg/KinovariSquad%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499138_456497179/330/cs3-1v4/s/v1/acmp/_WbB2KC7ZDs22xTYAFfwRyk0p3GfbW8L-q9exF4raO5H_YnoX1rdRmAGXSW9xQCk_7sxBLfA-WO0UT0ilQVPRVwo9GR6upY8VLlMWyRHtntEaNOnUakzaiUBcTIE5mGRYs3ti5Ek1JGmhtioUBPEr94GVtrOMeKGRPe4s56oYHWlO3xXrg/SLSMusic%20-%20unravel%20-%20Slow%20Version%20(From%3A%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001343575_62343575/235/cs3-4v4/s/v1/acmp/6pYoGLam7Gpf2Urd4PDhHInGLa0NZXgC4MiBNBCE_PG-YBTqjPZiIJNdgJty3HEEA42N4htrAN0TyudLT_l0rASk3yvwH8HSSl1GEA7xTxy8ZuQ1BfLrpHwIZ6g9MY2saaMmwaz51K32mUtBQHKXltyWuCvWzwYteRs3ZwuOhyEES2AJFQ/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499226_456421440/235/cs3-4v4/s/v1/acmp/ZHspQ1RQdnss3kdW6LREudZe1CVOFo2wItlxVeppU8SDGGPmfoVUH1p4451LnaaNG61RtRDrZD46eBfoTGwdmsiXmZ12Fzyhg_A5UMGplsTPHxOcL9Pw8CJ-VTQlIEwNiLJKwoWXxKIZ0e4h2A7rq3hRs49qnAzIc3lRE93yzrLCymc0ng/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499206_456447563/243/cs3-6v4/s/v1/acmp/26YcSjZs2EOKlDmgk6YspYb5imv6RXR1NsTkMBi6jgJMvdf5nNX3vJLWO-z9YAxl97b8JTxF3HK_u5wT7TcQfiIcFHOoLvXM7jg_Y1YMW1Vlc-sj-kqadoXFbGw_x9aI0NzZIT-VpYncsYC6N0N3cQsi_UjPPgHT5iiuIvFW6JHFBm5ROg/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel(%E3%82%AA%E3%83%AA%E3%82%B8%E3%83%8A%E3%83%AB%E3%82%A2%E3%83%BC%E3%83%86%E3%82%A3%E3%82%B9%E3%83%88%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E3%83%A1%E3%83%AD%E3%83%87%E3%82%A3%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz//download/474499222_456568196/98/psv4/s/v1/amp2/5MdAXyMQXscyhnNEfxrNMz4rZAhA3Ckw9tV0tLA94BDtvVCeDVz_Ycc62dFPtyJvjDaucoWBQZAguZHmkHV81InGjY22Am3TtoTYWH4rGOoiaM8coOi8DpIUQd21gda6fFcTZMkuLf0t33P5it1dH86nAmDGduot9AYu/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Winter).mp3?extra=null
https://slider.kz//download/474499266_456496174/238/cs3-2v4/s/v1/acmp/GkaHnOLSSRydiMt--oHvxPF2ZrdnAK-NO5Lei7J48RoxS_io3XwVnMFTQJN54MLZ5oagc9cSb4COMi_yxoCuAEm4FMnA3Kqetj5cj5D_alBQkHJEtNHS4IL1-kgE_QQ9RDsAfu5Qq2GrdyXj06rpvRr1EyKs1ITAkPez6uvlHAozli0mWQ/The%20Unknown%20Songbird%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/-2001091207_65091207/243/cs3-4v4/s/v1/acmp/izDJAIBhtw7Gbo37AL4HCeFiLWypB9BgtdIqUnziUZ06PBBWt6OVqGY5BJIjaY4E3fG1szeoZcQeU7946ltUPsX86cPyq9jMfaPTbNS2rk4qaDYoJtK70qlGrA_OwvHUS3Ec_BQTFNq3-fqv2tJuyElVIaUesQOo4MMH7BpaTUI1T3n-BA/Bibi%20Gamino%20-%20Unravel%20(Opening%20de%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001815387_86815387/189/cs3-5v4/s/v1/acmp/1q3ZcWs-DyWh-iYHwleI-jAD5A1WVgP8sN0zolnoNmBaj55OQqmHopLzZPd7AWmYP2Xs6E7Tv7BDtGn3TU1_-cDD13sbCywSWwQB2ckEsW84uZKn38qj0vcbYCN7YqDp7L4-drmoSdzIyHRE6xXrixB9cJuXgcKlTbSwkFh7-HRoU1lA9g/Baur%20Karbon%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001343001_91343001/91/cs9-18v4/s/v1/acmp/pk4LPfaKgfokG8pHZAzxFPGhpemNTcIw-Ey_8BN1RPX_QNtv9m7H5HzsrKVY-b2p1_AKiSm_jJOdWV1iUXGsjtOF-mkjo4er3adDmZSFB6BqlCGQgXkK7F1Kn8XQDc5dc41KESB5p-JTSPKOCzitje8M8wcsdELenOJANNsvKtByjHRL6A/Ren%20Avel%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/-2001459950_102459950/222/cs3-5v4/s/v1/acmp/dOeaiW4FJ3Q0jfOOTinST8c8Pn59ODRoRT5BgYrmn-DJw7m-u3x--lLPiUeECWRcHc7kKLGuAKoVTMUt32QfU0RDeGRCBGV2F8V8MPzNNczL3Qk6GQP9Bpj4ZL7SUUWlINmeRZyDChuD1APnUZtAzQ31DhAkQgpCH70MMHmtl6SNLNCVCQ/Jonathan%20Young%20-%20Unravel%20(Full%20Version).mp3?extra=null
https://slider.kz//download/474499249_456500220/95/cs9-5v4/s/v1/acmp/Cd6m3qhcIfjSesLIbaLc8IPOVc4E0H9ljKh_VIlj5LcIDifka-qMXBDquBCwd16bV2an-ZGX9X_NFFyV32q5DOg6AEKAMrSOdWQl9EC_mkvCLX1RLBF8HVarOoMZ9Y9oaV-gDNO9iTstvL0vBGQoxYz-CXIKok5FOfC5ptoG3dfSsRwkmA/Tony%20Gu%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BOrchestral%20Arrangement%5D.mp3?extra=null
https://slider.kz//download/-2001472007_90472007/95/cs9-2v4/s/v1/acmp/sqXJVOoZTIZvRB2CxjUHJZdvYHeaQkVffFqfpn4SW39S1-lImykJzWyENkb2tWUMO9eZNnOwNR0a6eZWPKvrBKdpLzLQE7JJFvV9kxKu8h2inNFyhSazlAljMtC3cn8R5Wk1B0uQdQY-17wb6-oBb7qoGSeiIBT-7C_Ob4G-JXcJ_6EUJw/Radio%20Control%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499153_456452837/238/cs3-6v4/s/v1/acmp/IQdnnyJQZtLM-cx27JQiYpqw3hrvc7Y4LGbUy5ruMBZNzC-7Bb4W6Wlw3M5CLykxFM2ALSU-aPR2xyMvggKW4gtqeXIhyfOq4yQbFNCYgG7mXXwwHp18FeyDTkXdYbdCbOZ05gQpO26e54LFLEky3FaAwvy48gQdHSUBAlCDNFN7gNUoMA/Kevin%20Remisch%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499122_456591314/97/cs3-7v4/s/v1/acmp/05S842jqQXODQOr-pq-LCi3z3CKmZR-6f9sqWNjGrXOaJDf-Ga0F42wMYDQmLTtChJaCKz-4kIdgGTXL-unYs4sJd3cg7ZwbywsPL2lMS5CiseljaXJ5I960mN1frRnZV-jVMiNqASMWfF9m5COgxgS_IAEvVDW5_tiXQxMxDfp_JSygOA/Lil%20Bubb1egum%20-%20unravel%20but%20is%20it%20okay%20if%20it's%20lofi%3F.mp3?extra=null
https://slider.kz//download/-2001077985_82077985/238/psv4/s/v1/amp2/TJu-2VmMhrc6VA7SF1OM2092kq003sX40joJxtIzXgAulLrdmWnzcVDMjn7L-dp5oXZVnqh-tXxyEbY-20YIvqOOWOuYlnldJHceTYDII9aJM-JhsA_Y9jkdCSOKZ0lwp_OhCWGFKEbfHu3NBJanFbms4EdBFUhNQH7K/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel%20(%E5%8E%9F%E6%9B%B2%E6%AD%8C%E6%89%8B%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz//download/474499138_456433311/238/cs3-6v4/s/v1/acmp/CqmMRJm5B03c3so1ex1GshI705Z472EzHgAWm-M9eznXzjVF_6G-7lYlhkHdfPpHsWcVU6hWhLRqVr6WrYKBXcyvZ-ri0q5iV9Jv8Iwyd91t9cLu0sWVP1W3016tZKK0CSoXyfiL_oOfFhHbF9DbqmjmyUVO9ZB-McrQY6PlmOk4M-MBHg/PelleK%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/-2001778472_74778472/238/cs9-2v4/s/v1/acmp/26oh4w3DDMsrZziue-jJNLoL_DKv7JMHYjXTixs0K4KjVjP0V6qTG2t5nmh6H9e8DVf3PH6Xoeqq1nzN718lXcKkSuutTFbWpgvA1nmV8M3DlQvm_WXyWRMReTYLwSwmwpALZ60-VbutNMFN3l8vPJ9F7lJCyi7V2ncQu9KxyWMCRD5C1Q/senakisaragi%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499220_456599167/246/psv4/s/v1/amp2/SX9vu47zGi9Uos8gQWm7XbfU1l0EKA1jqYblOj3ePK0pl5yPWLNFqITGYWgpzA1sRw6DzP_RAzgnu0C9-b2Ej5NDF4gU0px6dVSeOvO0DlKeW6XGcy2jFMJJCjHB_ag1VXEJgRb3jUYZLYRItNORbxOd45tRox8MGKBm/Matteo%20Leonetti%20-%20Unravel%20(Japanese%20Version)%20%5BTokyo%20Ghoul%5D.mp3?extra=null
https://slider.kz//download/-2001804556_66804556/238/cs3-5v4/s/v1/acmp/SGrWKb-wUFGqKV704k_bMGD7EJHh_A6sw2C9D8sWzWUA0R956jql_ZlVKEMv0MsJQ8gFsacMt_tywv3ZeXisu-rGTn4j_E8T3HS-o5RaqLzLOJauSlPoidthk8eIcx5xy4tGURb4KQkIfwIFJhriTy9HlnR7vCbaPpj_PfVNTBdlo7CU2g/V0RA%20-%20Unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz//download/474499291_456568508/102/cs9-6v4/s/v1/acmp/44hsI7dzI30x7Ho7N20vnYC06nWK2Sx-iXDPJXyN--Coele4vZUdekMfyNazmGRQ3IYMnLr1XBMPRWLVqZFkzHLk4fa_H5JN85IDDtft8EGPjsBRpSJOmbiAVAodQwQw3JcF6LW88ajmqKtJh3eUYARn9vrfQVN_J-ODVFY-dYm_akXUmg/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
https://slider.kz//download/-2001227120_98227120/186/psv4/s/v1/amp2/s57ppy-s9vm4n3qy-y4h8p4l0f5XyxK38-d91I3odmc9gvabMD_t42h8zd2oRL49tQkR1CD_vsPwIRA6DHsRUuW2WAkGJVwcef_VBmXUGsw7xEJSJG8kx-9AIXAUlj-8Ot32vMWg4TCiy/Olga%20Scheps%20-%20Start%20Anew%20(from%20%22Unravel%20Two%22).mp3?extra=null
https://slider.kz//download/474499232_456549884/90/psv4/s/v1/amp2/-eUtSpHonW9PPES9zCRzzNYbTSu7i_AEVujmMuifOKCuYme-q6vqFR5fXVHik2ARsaGX9_x4Za45VNNZohqsdkdCToDbne2W_6P7zgqnPJSuosGi8Z4ol0h0V_ViLH1ZIw_meOnJOkFDOTjKKyF-Ol0IvevzfvqPJtC0/Mary%20Antoine%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001667855_68667855/186/psv4/s/v1/amp2/IwKjMHOHdlAXWq1z1B5qxNBGhN8JVmJy-pJEl_b3RrTdZrGTPB_U0KIkNnnq7NcP5e1wa5ms9jSB1LIhEdiZTN6c7PHTMyJaLxNM8TDA46JpfBIXy73o11mf_FyBbAjkOIevfweKBRPukNSdConEqZr2ZUQoVaFeT7ZK/Track%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499146_456450407/186/psv4/s/v1/amp2/NMuGmiwJOxvdnOKQLkH0gIvMLbj_I_b5UIivo2Q2qVIxnI6kkJ5ITkC9rtL3H-28k3P6SzP-SEiegRZp_CyuXtuxPOA-qcx8Jo7VJEu817oIipyKTLFF9TBnMa6_E7w4Fa9uWM7RhR_J0VBAdPCDE4hi58gKNRBJ3bl5/Track%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499318_456592131/96/psv4/s/v1/amp2/zVQkmtvUwnxkcZxoMYAgcJ-RBoo8WvzKV6PyKAif3fw1C7fr-rzaWQbimUXK7_xSaKfsZpokZ7WRNdLOzxWm8QP8rlG2NNHYNQv6b2OmOiqOpP-WqeQ3VRvHN-0FtzcWnjYsIJZKcLcBA_q89U2c9Otw2Bqcvbt4lxRq/Vangakuz%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499173_456432829/137/cs3-1v4/s/v1/acmp/_LgsM2-WTNOvaV85Hy6ObbzZSKnJNfppdaWt9A_NyTv3XYhfSpyJ3QXFyQDb8nISAoRbTgNEt4yXtvvbWDli7gClVqaNCdiuTpEIwDl0EUZqyUF_aJrueZAjvd1DdUJR0YhvL6Fmv5C-8KoGEKotgiL1Eq1O4hU1wFDxayNVb4TkOzwwEQ/Kenzie%20Smith%20Piano%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001400485_76400485/187/psv4/s/v1/amp2/TFqk4JfH34feLtswrYZZXKfFUtIfXRTq8bOmXxLIHX99vXnkmPTp_A6-Rjj8fS9qiAQ6ao9F-VQvLZSH0bAnjBiUCM8pZbtBsL6z1v38vZzsc3kpP21MYG6kBCtfqg_jJhlj4CqDmfvQ0nW1NoUSNmutLvFy_7KTyjVG/Simpsonill%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001521828_57521828/551/cs3-6v4/s/v1/acmp/kXLXCQe33bHi2WDA_-0y9UGNCgbK7Xhca4q6pig_COLFf5bNaN11Z-4Kzd3A9tanob-U9_Jni_Ix5beoPUaJurH1ozqI6XQiKuHPmkxkLKRSXZpd_tj6SQhkniNDwxBxyRANOsxVfbHe80fcXahF0OWTKc4R5mjB1L1XVLsOhILcdpyS1g/Alveol%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499237_456436594/88/psv4/s/v1/amp2/yha2bLmL0XLvXvjfVKdqa25-JSAeZQYu8QQKRZKNwVAINm3912ZetkeVDrvSt5MeOyyto6x-PvbwC7hiBAXKC7g8_PwbcXKcqmku8hiHr7xJKl9ca34WIzeqNLyayp9RcFxGjO_Gk_AuaQ0ZBgXTaChq2nE4OTl1bqbX/MattxAJ%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001915683_101915683/90/psv4/s/v1/amp2/E79KiQTMA5b8mthMlpzYU5TRFcns8aqBNzqvKDFGJ14IR9Cz_C5yChPQUPQW-VFAm9_8sbo51D01NENhaegDJJIyui36C5ECVDwGaLhOcoLrhB8RKPLBGDLN01fcrb8mIxb7t0MhWr_YefPgj4Zt13xlufjor89n84eg/Music%20Legends%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499272_456394915/236/cs3-4v4/s/v1/acmp/2LiycWJnJntKnyikeZn_7TLKdSb8v343jFSIXZ9dbhcnsbSsjQZvlUI0jBkPWRGwGp5cK7UdprZarl3TUh_e_j_0vPrQs1SRHyogfbJlfQuM54Tr34iFNBasQLHCmZv1ktctdpEmuwNm4Yj2yi_1wLs9Iwqc9lBaZ-WmbDsfeeayQNwOLg/Shiroku%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001529426_101529426/255/cs3-2v4/s/v1/acmp/em6rSRlSNNcJHpGpiuSV4-VjdhJoAsemcgDXdS5VPu3XYPF1y-45PleMXDNAZ8jNEkf-5V9h7AInOtxaLBiP8KcWDvKW5d6EVy_hhh_qqK4PgpSPLS5Kv6yhdh7gUy2vo0t-ZVt-mZcQ1-8k48Q58_JE2U4JRD2ZgV8yVVbXKAK92bDnqg/HalcyonMusic%20-%20Unravel%20(from%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499176_456433059/241/psv4/s/v1/amp2/ATyS1Qxr7uBxM2KN4G0dMynLla7gPAaTh9nkDH5fMGYBEycOzy3sfM8LmLIjM2JP5yO3Vml0jBuTpeLoY9BBFCcWaD4qZ81dHjXsWFddJdyXqK0lkV4aKOo8EbMdyCJwRzeCesqn3r4aa_Pvxp3f6oPzLQgubvIXUF-s/I%20am%20Justice%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001621142_58621142/68/psv4/s/v1/amp2/CL5J0kD4Xm1iSLFGqD7zrODdTqubYTFwGCzC5r1n3Z_Tu6G_68RtjH3HaEY_D8qHuckrW0C0bCfUD5fXz0gxfoJD5PtEGwPpi_wQUdGi-D7dJITBjyMxx6dWz5duvvqZ7dmmW7ydKBGGaxHPI9_tofxRB_qqQmop_I11/Gianni%20Tallone%20Della%20Cavallotta%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001052300_100052300/89/cs9-11v4/s/v1/acmp/IvCOhCCclMK3tCeswQwVtNhtyYHX5lgjJUZfQsCfskiHciXLjBpj9SzjZMop3u-kPRSuyeyFvJoJEiP-2z_owq5MPdx4uNgrbl5Mqm0xbC_JWlpl4VH8Sc_vdNn28SHpU6EJiWDKgN7kf_sRozvO5OCEq7YmcfQe2lC_awFHsJV0KU7rHg/Fonzi%20M%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499157_456624310/212/psv4/s/v1/amp2/-oLu8sAswnvZ-I4RRXNqDu59CLZGoMUsv6qNVAL9QTVO9TcP4SWmGmUJSgfiXAw-_MgjRO1Y1y90YBuAirIcJgFenZe0XxlsqWrtbr6CEyQypX3sfmidTTBcH91G2x-4dXiDLI6JUmSXR9vZ4KQ_Z4vl-9o8SbiPmdD8/Love%2C%20Lies%20and%20Fiction%20-%20Unravel.mp3?extra=null
https://slider.kz//download/371745435_456449924/271/cs3-2v4/s/v1/acmp/ased-v3q4z-i_oiBH_OoVrZntdBkQGYKVQre0wr9DyNB7KVvbXTnpHh6G0_EyuBf90j3bW9SbB6aUeTt8NgAKKiuezgzETKKXzUhtpacQuKQEo-upEFNHLq5VzSrFAz2IjSSA3lm-p9plN7U4DQOPnCzn1_klu8Mje4uY3TCcwoTAuPckg/The%20Blue%20Notes%20-%20Unravel%20from%20%22tokyo%20Ghoul%22.mp3?extra=null
https://slider.kz//download/-2001260329_90260329/94/cs9-15v4/s/v1/acmp/qI5QjSZzHW-fYGXdVihZndt972zUCvmUN2Qzjk9gfUgyTjh_XPeOh9ctff9Ob2ezMEbFLEL7oNkrnvgoh4EJy9JlUjJPhTQS2Fh3L1cLzqJELD-waEz9vDaZwS_IuPXr0nH2PO5img_a4xkkqRLIy-VDm64EVAdGelBPmwLf7rIj_qjF9w/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
Using requests module from api
import requests
import json
headers={
'X-Requested-With': 'XMLHttpRequest'}
api_url='https://slider.kz/similar/artist/unravel'
req=requests.get(api_url,headers=headers).json()
for url in req['similarartists']['artist']:
print(url['url'])
Output:
https://www.last.fm/music/Contaminated
https://www.last.fm/music/Excarnate
https://www.last.fm/music/Extremity
https://www.last.fm/music/Serpent+Of+Gnosis
https://www.last.fm/music/Petrification
https://www.last.fm/music/Skullsmasher
https://www.last.fm/music/Hissing
https://www.last.fm/music/Mortuous
https://www.last.fm/music/Skelethal
https://www.last.fm/music/stabbed
https://www.last.fm/music/Winds+of+Leng
https://www.last.fm/music/Usurpress
https://www.last.fm/music/Encoffinized
https://www.last.fm/music/Glacial+Tomb
https://www.last.fm/music/Overlord
https://www.last.fm/music/RESIN+TOMB
https://www.last.fm/music/Descent
https://www.last.fm/music/Hurricane+Death
https://www.last.fm/music/Gutter+Instinct
https://www.last.fm/music/Crypt+Crawler
https://www.last.fm/music/This+Game
https://www.last.fm/music/LIFETAKER
https://www.last.fm/music/Autokrator
https://www.last.fm/music/Vanhelgd
https://www.last.fm/music/ATARAXY
I'm trying to scrape website traffic from semrush.com.
my current code using BeautifulSoup is:
from bs4 import BeautifulSoup, BeautifulStoneSoup
import urllib
import json
req = urllib.request.Request('https://www.semrush.com/info/burton.com', headers={'User-Agent':'Magic Browser'})
response = urllib.request.urlopen(req)
raw_data = response.read()
response.close()
soup = BeautifulSoup(raw_data)
I've been trying data = soup.findAll("a", {"href":"/info/burton.com+(by+organic)"}) or data = soup.findAll("span", {"class":"sem-report-counter"}) without much luck.
I can see the numbers on the webpage that I would like to get. Is there a way to pull this information off? I'm not seeing it in the html I pull.
I went the extra mile and set up a working example of how you can use selenium to scrape that page. Install selenium and try it out!
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url = 'https://www.semrush.com/info/burton.com' #your url
options = Options() #set up options
options.add_argument('--headless') #add --headless mode to options
driver = webdriver.Chrome(executable_path='/opt/ChromeDriver/chromedriver',
chrome_options=options)
#note: executable_path will depend on where your chromedriver.exe is located
driver.get(url) #get response
driver.implicitly_wait(1) #wait to load content
elements = driver.find_elements_by_xpath(xpath='//a[#href="/info/burton.com+(by+organic)"]') #grab that stuff you wanted?
for e in elements: print(e.get_attribute('text').strip()) #print text fields
driver.quit() #close the driver when you're done
Output that I see in my terminal:
356K
6.5K
59.3K
$usd305K
Organic keywords
Organic
Top Organic Keywords
View full report
Organic Position Distribution
import requests
import pdfkit
# start a session
s = requests.Session()
data = {'username': 'name', 'password': 'pass'}
# POST request with cookies
s.post('https://www.facebook.com/login.php', data= data)
url = 'https://www.facebook.com'
# navigate to page with cookies set
options = {'cookie': s.cookies.items(), 'javascript-delay': 1000}
pdfkit.from_url(url, 'file.pdf', options= options)
I'm trying to automate the process of saving a login-protected webpage as a PDF by setting the cookies and navigating to the page using requests. Is there a better way to tackle this/something I'm doing wrong?
Portal sends login and password with different names and also sends hidden values which can change in every request. It sends to different url than login.php and it can check headers to block bots/scripts.
It can be easier with Selenium which control browser and you can take picture or get HTML to generate PDF.
import selenium.webdriver
import pdfkit
#import time
driver = selenium.webdriver.Chrome()
#driver = selenium.webdriver.Firefox()
driver.get('https://www.facebook.com/login.php')
#time.sleep(1)
driver.find_element_by_id('email').send_keys('your_login')
driver.find_element_by_id('pass').send_keys('your_password')
driver.find_element_by_id('loginbutton').click()
#time.sleep(2)
driver.save_screenshot('output.png') # only visible part
#print(driver.page_source)
pdfkit.from_string(driver.page_source, 'file.pdf')
Maybe using driver "PhantomJS" or module PIL/pillow you could get full page as screenshot.
See generate-full-page-screenshot-in-chrome
With wkhtmltopdf, you can do something like this from command line:
wkhtmltopdf --cookie-jar cookies.txt https://example.com/loginform.html --post 'user_id' 'my_id' --post 'user_pass' 'my_pass --post 'submit_btn' 'submit' throw_away.pdf
wkhtmltopdf --cookie-jar cookies.txt https://example.com/securepage.html keep_this_one.pdf
I'm try to write a web crawler that download a CSV file by a dynamic url.
The url is like http://aaa/bbb.mcv/Download?path=xxxx.csv
I put this url to my chrome browser but I just start to download immediately and the page won't change.
I can't even find any request in develop screen.
I've tried to ways to get the file
put the url in selenium
driver.get(url)
try to get file by requests lib
requests.get(url)
Both didn't work...
Any advice?
Output of two ways:
I try to get the screen shot and it seems doesn't change the page. (just like in chrome)
I try to print out the data I get and it seems like as html file.
Then open it in the browser it is a login page.
import requests
url = '...'
save_location = '...'
session = requests.session()
response = session.get(url)
with open(save_location, 'wb') as t:
for chunk in response.iter_content(1024):
t.write(chunk)
Thanks for everyone's help!
I finally find the problem is that...
I login the website by selenium and I use requests to download the file.
Selenium doesn't have any authentication information!
So my solution is get the cookies by selenium first.
Then send it into the requests!
Here is my Code
cookies = driver.get_cookies() #selenium web driver
s = requests.Session()
for cookie in cookies:
s.cookies.set(cookie['name'], cookie['value'])
response = s.get(url)
I have a page that has a table (table id= "ctl00_ContentPlaceHolder_ctl00_ctl00_GV" class="GridListings" )i need to scrape.
I usually use BeautifulSoup & urllib for it,but in this case the problem is that the table takes some time to load ,so it isnt captured when i try to fetch it using BS.
I cannot use PyQt4,drysracpe or windmill because of some installation issues,so the only possible way is to use Selenium/PhantomJS
I tried the following,still no success:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS()
driver.get(url)
wait = WebDriverWait(driver, 10)
table = wait.until(EC.presence_of_element_located(By.CSS_SELECTOR, 'table#ctl00_ContentPlaceHolder_ctl00_ctl00_GV'))
The above code doesnt give me the desired contents of the table.
How do i go about achieveing this???
You can get the data using requests and bs4,, with almost if not all asp sites there are a few post params that always need to be provided like __EVENTTARGET, __EVENTVALIDATION etc.. :
from bs4 import BeautifulSoup
import requests
data = {"__EVENTTARGET": "ctl00$ContentPlaceHolder$ctl00$ctl00$RadAjaxPanel_GV",
"__EVENTARGUMENT": "LISTINGS;0",
"ctl00$ContentPlaceHolder$ctl00$ctl00$ctl00$hdnProductID": "139",
"ctl00$ContentPlaceHolder$ctl00$ctl00$hdnProductID": "139",
"ctl00$ContentPlaceHolder$ctl00$ctl00$drpSortField": "Listing Number",
"ctl00$ContentPlaceHolder$ctl00$ctl00$drpSortDirection": "A-Z, Low-High",
"__ASYNCPOST": "true"}
And for the actual post, we need to add a few more values to out post data:
post = "https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx"
with requests.Session() as s:
s.headers.update({"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"})
soup = BeautifulSoup(s.get(post).content)
data["__VIEWSTATEGENERATOR"] = soup.select_one("#__VIEWSTATEGENERATOR")["value"]
data["__EVENTVALIDATION"] = soup.select_one("#__EVENTVALIDATION")["value"]
data["__VIEWSTATE"] = soup.select_one("#__VIEWSTATE")["value"]
r = s.post(post, data=data)
soup2 = BeautifulSoup(r.content)
table = soup2.select_one("div.GridListings")
print(table)
You will see the table printed when you run the code.
If you want to scrap something, it will be nice first to install a web debugger ( Firebug for Mozilla Firefox for example) to watch how the website you want to scrap is working.
Next, you need to copy the process of how the website is connecting to backoffice
As you said, the content that you want to scrap is being loaded asynchronously (only when the document is ready)
Assuming the debugger is running and also you have refreshed the page, you will see on the network tab the following request:
POST https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx
The final process flow to reach your goal will be:
1/ Use requests python module
2/ Open a requests session to the index page website site (with cookies handling)
3/ Scrap all the input for the specific POST form request
4/ Build a POST parameter DICT containing all inputs & value fields scrapped in the previous step + adding some specific fixed params.
5/ POST the request (with required data)
6/ Use finally BS4 module (as usual) to soup the answered html to scrap your data
Please see bellow a working code:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
from bs4 import BeautifulSoup
import requests
base_url="https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx"
#create requests session
s = requests.session()
#get index page
r=s.get(base_url)
#soup page
bs=BeautifulSoup(r.text)
#extract FORM html
form_soup= bs.find('form',{'name':'aspnetForm'})
#extracting all inputs
input_div = form_soup.findAll("input")
#build the data parameters for POST request
#we add some required <fixed> data parameters for post
data={
'__EVENTARGUMENT':'LISTINGS;0',
'__EVENTTARGET':'ctl00$ContentPlaceHolder$ctl00$ctl00$RadAjaxPanel_GV',
'__EVENTVALIDATION':'/wEWGwKis6fzCQLDnJnSDwLq4+CbDwK9jryHBQLrmcucCgL56enHAwLRrPHhCgKDk6P+CwL1/aWtDQLm0q+gCALRvI2QDAKch7HjBAKWqJHWBAKil5XsDQK58IbPAwLO3dKwCwL6uJOtBgLYnd3qBgKyp7zmBAKQyTBQK9qYAXAoieq54JAuG/rDkC1djKyQMC1qnUtgoC0OjaygUCv4b7sAhfkEODRvsa3noPfz2kMsxhAwlX3Q=='
}
#we add some <dynamic> data parameters
for input_d in input_div:
try:
data[ input_d['name'] ] =input_d['value']
except:
pass #skip unused input field
#post request
r2=s.post(base_url,data=data)
#write the result
with open("post_result.html","w") as f:
f.write(r2.text.encode('utf8'))
Now, please get a look at "post_result.html" content and you will find the data !
Regards