We Keep Coding

How to exit and then automatically restart code

I have some python code which scrapes a website and reports the live price of a specific crypto. When I use a while loop to keep printing the live price it keeps printing the same price over and over even when the live price on the website has changed. I thought that maybe my code was scraping it and coming to that website too fast so I added a delay using the time module but even after a 1 minute delay it will not display the correct price but instead prints the same price over and over. Manually ending and restarting the code seemed to make this bug go away but I want this program to run 24/7 and email me when a price reaches a certain point. This is my code so far: (BTW I am a beginner)
import requests
import bs4
import time
run = True
while run == True:
# time.sleep(60)
res = requests.get("https://coinmarketcap.com/currencies/gitcoin/")
soup_obj = bs4.BeautifulSoup(res.text, "lxml")
item = soup_obj.select(".priceValue___11gHJ")[0]
item = item.text
print(item)
exit()
This has a loop but I have added an exit() function so that it ends and so I can manually restart it. I just need a way for this code to automatically end itself and then restart repeatedly. I am also using the community edition of Pycharm (latest edition).

You can write your program to call a subprocess instead of doing the web call itself. That subprocess can call requests, return whatever you want via stdout and exit. There are multiple ways to do this. You could write separate scripts or use multiprocess.Process, but in this example I've written a script that calls itself and uses command line parameters to know which role it is playing.
import sys
if len(sys.argv) == 1:
# run poller as subprocess so it exits
import time
import subprocess as subp
while True:
result = subp.run([sys.executable, __file__, "called"], capture_output=True)
# assuming program returns ascii float in single line
item = result.stdout.decode("ascii").strip()
print(item)
time.sleep(60)
else:
import requests
import bs4
res = requests.get("https://coinmarketcap.com/currencies/gitcoin/")
soup_obj = bs4.BeautifulSoup(res.text, "lxml")
item = soup_obj.select(".priceValue___11gHJ")[0]
item = item.text
sys.stdout.write(item)

How do I use the requests module within Pycharm?

I'm new to Python, and I'm working in Pycharm to read data line by line from a webpage. For this task, I'm attempting to use the requests module. However, when I try to print the response object, I see "Process finished with exit code 0" and no object displayed.
Do I need to create some sort of setting to be able to work with HTTP requests in Python?
Code:
import re
import requests
def find_phone_number(url='https://www.python-course.eu/barneyhouse.txt'):
response = requests.get(url)
return response
print(find_phone_number(url='https://www.python-course.eu/barneyhouse.txt'))

You need to call the function and access the 'text' element.
Also, in your code the print statement is not indented properly so it will never be run.
Here is an example of the code doing what I think you intendend:
import re
import requests
def find_phone_number(url='https://www.python-course.eu/simpsons_phone_book.txt'):
response = requests.get(url)
return response
text_you_want = find_phone_number().text
print(text_you_want)

Well, for starters, your find_phone_number() function calls itself after it returns. This is because your last line is indented and therefore inside the function definition. The reason you keep getting Process finished with exit code 0 is because your function is never actually called. This should work:
import re
import requests
def find_phone_number(url='https://www.python-course.eu/barneyhouse.txt'):
response = requests.get(url)
return response
print(find_phone_number(url='https://www.python-course.eu/barneyhouse.txt'))

How do i understand whether i am parsing the websites acurately?

I built this function to tell me whether there have been changes to the website. I'm not sure if it works as I have tried it on a few websites that have not changed and it has given me the wrong output. Where is the issue and is there an issue at all?
This is the code:
I put the code into a function so that I could allow the user to input any site
userurl=input("Please enter a valid url")
def checksite(userurl):
change=False
import time
import urllib.request
import io
u = urllib.request.urlopen(userurl)
webContent1 = u.read()
time.sleep(60)
u = urllib.request.urlopen(userurl)
webContent2 = u.read()
if webContent1 == webContent2:
print("Everything is normal")
elif webContent1 !=webContent2:
print("Warning, there has been a change to the webite!")
change=True
return change
checksite(userurl)

Try making a small HTML Hello World page. Given that many websites have dynamic content that changes each time you access it (and might not necessarily be visible), that could lead to your "incorrect" results.

I have tested your code and it works perfectly fine in a Python webserver.
I have started one with
python -m http.server
and placed an index.html in the same directory with some content before starting the server.
and your code
import time
import urllib.request
import io
userurl='http://localhost:8000/index.html'
def checksite(userurl):
change=False
u = urllib.request.urlopen(userurl)
webContent1 = u.read()
print(webContent1)
time.sleep(15)
u = urllib.request.urlopen(userurl)
webContent2 = u.read()
print(webContent2)
if webContent1 == webContent2:
print("Everything is normal")
elif webContent1 !=webContent2:
print("Warning, there has been a change to the webite!")
change=True
return change
checksite(userurl)
and output
b'<html>\n\t<title> Hello </title>\n\t<body>\n\t\tTesting, Webcontent1 \n\t</body>\n\t</html>\n\n'
b'<html>\n\t<title> Hello </title>\n\t<body>\n\t\tTesting, Webcontent2\n\t</body>\n\t</html>\n\n'
Warning, there has been a change to the webite!
[Finished in 17.5s]
Your code is perfectly fine.

to know if a website or a page has changed you need to have a backup of it somewhere, in your code it was like you were comparing the site to itself... anyways. i recomend using the requests library in addition to BS4 and try parsing it line by line comparing to the backup you have.
So while the code is working (aka: the site you have as backup is showing the same lines as the site on the web) it will have a variable true. if it has changed it breaks the loop and simply shows the line where the site has changed.

Module urllib.request not getting data

I am trying to test this demo program from lynda using Python 3. I am using Pycharm as my IDE. I already added and installed the request package, but when I run the program, it runs cleanly and shows a message "Process finished with exit code 0", but does not show any output from print statement. Where am I going wrong ?
import urllib.request # instead of urllib2 like in Python 2.7
import json
def printResults(data):
# Use the json module to load the string data into a dictionary
theJSON = json.loads(data)
# now we can access the contents of the JSON like any other Python object
if "title" in theJSON["metadata"]:
print(theJSON["metadata"]["title"])
# output the number of events, plus the magnitude and each event name
count = theJSON["metadata"]["count"];
print(str(count) + " events recorded")
# for each event, print the place where it occurred
for i in theJSON["features"]:
print(i["properties"]["place"])
# print the events that only have a magnitude greater than 4
for i in theJSON["features"]:
if i["properties"]["mag"] >= 4.0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"])
# print only the events where at least 1 person reported feeling something
print("Events that were felt:")
for i in theJSON["features"]:
feltReports = i["properties"]["felt"]
if feltReports != None:
if feltReports > 0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"], " reported " + str(feltReports) + " times")
# Open the URL and read the data
urlData = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_day.geojson"
webUrl = urllib.request.urlopen(urlData)
print(webUrl.getcode())
if webUrl.getcode() == 200:
data = webUrl.read()
data = data.decode("utf-8") # in Python 3.x we need to explicitly decode the response to a string
# print out our customized results
printResults(data)
else:
print("Received an error from server, cannot retrieve results " + str(webUrl.getcode()))

Not sure if you left this out on purpose, but this script isn't actually executing any code beyond the imports and function definition. Assuming you didn't leave it out on purpose, you would need the following at the end of your file.
if __name__ == '__main__':
data = "" # your data
printResults(data)
The check on __name__ equaling "__main__" is just so your code is only executing when the file is explicitly run. To always run your printResults(data) function when the file is accessed (like, say, if its imported into another module) you could just call it at the bottom of your file like so:
data = "" # your data
printResults(data)

I had to restart the IDE after installing the module. I just realized and tried it now with "Run as Admin". Strangely seems to work now.But not sure if it was a temp error, since even without restart, it was able to detect the module and its methods.

Your comments re: having to restart your IDE makes me think that pycharm might not automatically detect newly installed python packages. This SO answer seems to offer a solution.
SO answer

Adapting Python Code to Extract Mentions from Tweets for Mention Network

Background:
I found this code from Alex Hanna (https://github.com/alexhanna) and I've been trying to adapt it to my situation, but haven't been as successful as I would like, and was hoping for help.
This code takes JSON output from the Twitter API and extracts user mentions from it to create a mention network. Currently, the code is executed as follows:
cat file.json | mentionMapper.py
Desires:
I would like to change it so that the python code reads in the JSON file. Previously I've used the code below, but haven't been able to figure out how to adapt it for this instance.
for line in in_file:
try:
tweet = json.loads(line)
except:
pass
I would also like the code to write the output to a CSV file instead of printing it to the screen or having to manually direct the output using > after the code.
Ultimate Goal:
Edgelist of Twitter users who mention one another.
Code:
import json
import sys
def main():
for line in sys.stdin:
line = line.strip()
data = ''
try:
data = json.loads(line)
except ValueError as detail:
continue
if not (isinstance(data, dict)):
## not a dictionary, skip
pass
elif 'delete' in data:
## a delete element, skip for now.
pass
elif 'user' not in data:
## bizarre userless edge case
pass
else:
if 'entities' in data and len(data['entities']['user_mentions']) > 0:
user = data['user']
user_mentions = data['entities']['user_mentions']
for u2 in user_mentions:
print ",".join([
user['screen_name'],
u2['screen_name'],
"1"
])
if __name__ == '__main__':
main()