We Keep Coding

Get the specific response parameter with urllib in python

I am able to perform a web request and get back the response, using urllib.
from urllib import request
from urllib.parse import urlencode
response = request.urlopen(req, data=login_data)
content = response.read()
I get back something like b'{"token":"abcabcabc","error":null}'
How will i be able to parse the token information?

You can use the json module to load the binary string data and then access the token property:
token = json.loads(bin_data)['token']

Getting specific data from a website with python

code:
import requests
from bs4 import BeautifulSoup
url="https://covid19.saglik.gov.tr/"
R=requests.get(url)
print(R.text)
Question: Hello friends,I must receive below specific values from above website. These values are changing daily.When the program runs, it should be able to print out the specified key from the website .For example: print(data["tarih"]) , print(data["gunluk_test"]) , print(data["gunluk_vaka"] etc. in html script. How can I do that ?
CODE's OUTPUT RESULT:
var sondurumjson = [{"tarih":"13.05.2021","gunluk_test":"201.295","gunluk_vaka":"11.534","gunluk_hasta":"1.217","gunluk_vefat":"238","gunluk_iyilesen":"55.472","toplam_test":"50.259.943","toplam_hasta":"5.083.996","toplam_vefat":"44.059","toplam_iyilesen":"4.856.763","toplam_yogun_bakim":"","toplam_entube":"","hastalarda_zaturre_oran":"4,0","agir_hasta_sayisi":"2.765","yatak_doluluk_orani":"43,7","eriskin_yogun_bakim_doluluk_orani":"65,0","ventilator_doluluk_orani":"32,4","ortalama_filyasyon_suresi":"","ortalama_temasli_tespit_suresi":"8","filyasyon_orani":"99,9"}];//]]>

It looks like the dict is already recognised by python so json.loads fails did you paste the output of the api or did you print the dict that you are working on?
Anyhow if that is the result of your api you can just do this to access everything in the dictionary:
sondurumjson = [{"tarih":"13.05.2021","gunluk_test":"201.295","gunluk_vaka":"11.534","gunluk_hasta":"1.217","gunluk_vefat":"238","gunluk_iyilesen":"55.472","toplam_test":"50.259.943","toplam_hasta":"5.083.996","toplam_vefat":"44.059","toplam_iyilesen":"4.856.763","toplam_yogun_bakim":"","toplam_entube":"","hastalarda_zaturre_oran":"4,0","agir_hasta_sayisi":"2.765","yatak_doluluk_orani":"43,7","eriskin_yogun_bakim_doluluk_orani":"65,0","ventilator_doluluk_orani":"32,4","ortalama_filyasyon_suresi":"","ortalama_temasli_tespit_suresi":"8","filyasyon_orani":"99,9"}]
new_object = sondurumjson[0]
print(new_object['tarih'])

I think you should use R.json() instead of R.text

import requests
from bs4 import BeautifulSoup
import json
url="https://covid19.saglik.gov.tr/"
R=requests.get(url)
soup=BeautifulSoup(R.content,"html.parser" )
script=soup.find_all("script")[18]
s=str(script).split('\nvar sondurumjson = ')
#print(s[1])
a=str(s[1])
b=a.split("\r")
#print(b[0])
c=b[0].partition(';')
data=c[0]
#print(data)
data = json.loads(data)
print( "Tarih:", data[0]['tarih'])
print("Günlük Test:", data[0]['gunluk_test'])
print( "Günlük Vaka:", data[0]['gunluk_vaka'])

How to use Python requests to fill out a date option as well as download from a radiobutton

I'm trying to make a python script that will scrape data from this website
https://noms.wei-pipeline.com/reports/ci_report/launch.php?menuitem=2600315
and download the CSV from yesterday. As you can see, it's got two menu options for dates, a radio button for CSV and then a submit button.
I thought perhaps I could use the requests library? Not looking for someone to do it for me, but if anyone could point me in the right direction that would be great!
I know this is too simple but here is what I have so far:
import requests
print('Download Starting...')
url = 'https://noms.wei-pipeline.com/reports/ci_report/launch.php?menuitem=2600315'
r = requests.get(url)
filename = url.split('/')[-1] # this will take only -1 splitted part of the url
with open(filename,'wb') as output_file:
output_file.write(r.content)
print('done')

You need first to use requests.Session() in order to store cookies and re-send them in subsequent requests. The process is the following :
get the original URL first to get the cookies (session id)
make a request on POST /reports/ci_report/server/request.php with some parameters including date and output format. The result is a json with an id like this :
{'jrId': 'jr_13879611'}
make a request on GET /reports/ci_report/server/streamReport.php?jrId=jr_13879611 which gives the csv data
There is a parameter in the POST request where we need the menuitem query param value from your original url, so we parse the query params to get it using urlparse :
import requests
import time
import urllib.parse as urlparse
from urllib.parse import parse_qs
from datetime import datetime,timedelta
yesterday = datetime.now() - timedelta(1)
yesterday_date = f'{yesterday.strftime("%d")}-{yesterday.strftime("%B")[:3]}-{yesterday.strftime("%Y")}'
original_url = "https://noms.wei-pipeline.com/reports/ci_report/launch.php?menuitem=2600315"
parsed = urlparse.urlparse(original_url)
target_url = "https://noms.wei-pipeline.com/reports/ci_report/server/request.php"
stream_report_url = "https://noms.wei-pipeline.com/reports/ci_report/server/streamReport.php"
s = requests.Session()
# load the cookies
s.get(original_url)
#get id
r = s.post(target_url,
params = {
"request.preventCache": int(round(time.time() * 1000))
},
data = {
"ReportProc": "CIPR_DAILY_BULLETIN",
"p_ci_id": parse_qs(parsed.query)['menuitem'][0],
"p_opun": "PL",
"p_gas_day_from": yesterday_date,
"p_gas_day_to": yesterday_date,
"p_output_option": "CSV"
})
r = s.get(stream_report_url, params = r.json())
print(r.text)
Try this on repl.it

Python 3 requests.get().text returns unencoded string

Python 3 requests.get().text returns unencoded string.
If I execute:
import requests
request = requests.get('https://google.com/search?q=Кто является президентом России?').text.lower()
print(request)
I get kind of this:
Кто является презид
I've tried to change google.com to google.ru
If I execute:
import requests
request = requests.get('https://google.ru/search?q=Кто является президентом России?').text.lower()
print(request)
I get kind of this:
d0%9a%d1%82%d0%be+%d1%8f%d0%b2%d0%bb%d1%8f%d0%b5%d1%82%d1%81%d1%8f+%d0%bf%d1%80%d0%b5%d0%b7%d0%b8%d0%b4%d0%b5%d0%bd%d1%82%d0%be%d0%bc+%d0%a0%d0%be%d1%81%d1%81%d0%b8%d0
I need to get an encoded normal string.

You were getting this error because requests was not able to identify the correct encoding of the response. So if you are sure about the response encoding then you can set it like the following:
response = requests.get(url)
response.encoding --> to check the encoding
response.encoding = "utf-8" --> or any other encoding.
And then get the content with .text method.

I fixed it with urllib.parse.unquote() method:
import requests
from urllib.parse import unquote
request = unquote(requests.get('https://google.ru/search?q=Кто является президентом России?').text.lower())
print(request)

Using Urllib2 with 511 api that requires token

So, all I want to do is send a request to the 511 api and return the train times from the train station. I can do that using the full url request, but I would like to be able to set values without paste-ing together a string and then sending that string. I want to have the api return the train times for different stations. I see other requests that use headers, but I don't know how to use headers with a request and am confused by the documentation.
This works...
urllib2.Request("http://services.my511.org/Transit2.0/GetNextDeparturesByStopCode.aspx?token=xxxx-xxx&stopcode=70142")
response = urllib2.urlopen(request)
the_page = response.read()
I want to be able to set values like this...
token = xxx-xxxx
stopcode = 70142
url = "http://services.my511.org/Transit2.0/GetNextDeparturesByStopCode.aspx?"
... and then put them together like this...
urllib2.Request(url,token, stopcode)
and get the same result.

The string formatting documentation would be a good place to start to learn more about different ways to plug in values.
val1 = 'test'
val2 = 'test2'
url = "https://www.example.com/{0}/blah/{1}".format(val1, val2)
urllib2.Request(url)

The missing piece is "urllib" needs to be used along with "urllib2". Specifically, the function urllib.urlencode() returns the encoded versions of the values.
From the urllib documentation here
import urllib
query_args = { 'q':'query string', 'foo':'bar' }
encoded_args = urllib.urlencode(query_args)
print 'Encoded:', encoded_args
url = 'http://localhost:8080/?' + encoded_args
print urllib.urlopen(url).read()
So the corrected code is as follows:
import urllib
import urllib2
token = xxx-xxxx
stopcode = 70142
query_args = {"token":token, "stopcode":stopcode}
encoded_args = urllib.urlencode(query_args)
request = urllib2.Request(url+encoded_args)
response = urllib2.urlopen(request)
print(response.read())
Actually, it is a million times easier to use requests package and not urllib, urllib2. All that code above can be replaced with this:
import requests
token = xxx-xxxx
stopcode = 70142
query_args = {"token":token, "stopcode":stopcode}
r = request.get(url, params = query_args)
r.text