We Keep Coding

Scraping chat file: How to automatically add missing usernames

I have a local html file containing messages from a telegram chat.
I scraped each message and extracted information like ID of the message, date it was posted and the content of each message.
The Problem: If a user posts multiple times in a row, only the first message will contain the user's name. After that, the name-column in the df remains empty- Because: All the following messages have their own ID and time stamp – but they do not have the div class="from_name" information.
Is there a way to fix that so that each message has a user name information in the df?
messages = doc.select('div.message')
rows = []
for message in messages:
print('---')
row = {}
row['id_number'] = (message['id'])
try:
row['time'] = (message.select_one('div[title]').get('title'))
except:
print("Couldn't find the time")
try:
row['username'] = (message.select_one('div.from_name').contents[0].strip())
except:
print("Couldn't find a name")
try:
row['text of the message'] = (message.select_one('div.text').text.strip())
except:
print("Couldn't find a text")
print(row)
rows.append(row)

You can handle it in your DataFrame or explicitly in your code - Instead the try/except blocks you can check directly if element is available, if not set the username from last added row in rows:
if message.select_one('div.from_name'):
row['username'] = message.select_one('div.from_name').contents[0].strip()
else:
row['username'] = rows[-1].get('username')
print("Couldn't find a name, set last username")
EDIT: Based on your comments you could check len(rows) and continue with next iteration if it is empty:
if message.select_one('div.from_name'):
row['username'] = message.select_one('div.from_name').contents[0].strip()
elif len(rows) > 0:
row['username'] = rows[-1].get('username')
else:
continue

How to fix the Python Selenium CSV file loop problem?

I'm trying to create a script for loop from the CSV file to filling up the website email and password.
But the loop is only stuck at the first ROW, can't go to the next ROW. Anyone can help?
Open Browser
Go to a website
Auto Log-in (filling up email and password details from csv file )
Close Browser
Re open again the website(loop start from the second step )
Re Auto log-in but with the second account (filling up details from csv file SECOND ROW ) .
Re do the same tasks 50 times (From account 1 to 50 for example )
I am running Python 3.8.1 and PyCharm.
import pandas as pd
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome = webdriver.Chrome(options=chrome_options)
chrome.get("https://facebook.com")
data = pd.read_csv(r'D:\PyCharm\test.csv')
df = pd.DataFrame(data)
i = 0
while i == 0:
a = 0
Username = df.username
Password = df.password
id_box = chrome.find_element_by_id('email')
id_box.send_keys(Username[a])
Pass_box = chrome.find_element_by_id('pass')
Pass_box.send_keys(Password[a])
Send = chrome.find_element_by_name('login')
Send.click()
try:
test = chrome.find_element_by_id('pass')
id_box.clear()
Pass_box.clear()
except:
print("logged in")
break
a = a + 1

Make tweepy search for the newest mentions instead of the oldest

Today I wrote a twitter bot that replies anybody who mentions it with a random image from a folder.
The problem here is that I'm a newbie in python and I don't know how to make it funcitonal at all. When I started running it, the bot started replying all the mentions from other users (I'm using an old account a friend gave to me), and that's not precisely what I want, even if it's working, but not as I desire.
The bot is replying all the mentions from the very beggining and it won't stop until all these are replied (the bot is now turned off, I don't want to annoy anybody)
How can I make it to only reply to latest mentions instead of the first ones?
here's the code:
import tweepy
import logging
from config import create_api
import time
import os
import random
from datetime import datetime
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
api = create_api()
imagePath = random.choice(os.listdir("images/"))
while True:
for tweet in tweepy.Cursor(api.mentions_timeline).items():
try:
imagePath = random.choice(os.listdir("images/"))
tweetId = tweet.user.id
username = tweet.user.screen_name
api.update_with_media('images/' + imagePath, "#" + username + " ", in_reply_to_status_id=tweet.id)
print('Replying to ' + username + 'with ' + imagePath)
except tweepy.TweepError as e:
print(e.reason)
except StopIteration:
break
time.sleep(12)
Thanks in advance.

I don't have the ability to test this code currently but this should work.
Instead of iterating over every tweet, it turns the iterator that tweepy.Cursor returns into a list and then just gets the last item in that list.
api = create_api()
imagePath = random.choice(os.listdir("images/"))
while True:
tweet_iterator = tweepy.Cursor(api.mentions_timeline).items()
latest_tweet = list(tweet_iterator)[-1]
try:
imagePath = random.choice(os.listdir("images/"))
tweetId = latest_tweet.user.id
username = latest_tweet.user.screen_name
api.update_with_media('images/' + imagePath, "#" + username + " ", in_reply_to_status_id=latest_tweet.id)
print('Replying to ' + username + 'with ' + imagePath)
except tweepy.TweepError as e:
print(e.reason)
except StopIteration:
break
time.sleep(12)
You will also want to keep track of what user you last replied to, so you don't just keep spamming the same person over and over.
This isn't the most efficient way of doing it but should be easy enough to understand:
latest_user_id = None
while True:
# Rest of the code
try:
if latest_user_id == latest_tweet.user.id:
# don't do anything
else:
latest_user_id = latest_tweet.user.id
# the rest of your code

How to prevent my Selenium/Python program from crashing?

I made a program which gets one record from Google Sheet process on it then delete it and so on. If I update Google Sheet then the program will deduct record in the next loop and process on it and then delete,
but it runs only 1 or 2 hours and then program gives an error:
What can I add in my program so my program never stops?
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
import traceback
import string
import gspread
from oauth2client.service_account import ServiceAccountCredentials
from selenium.common.exceptions import NoAlertPresentException
from selenium.common.exceptions import UnexpectedAlertPresentException
Email=raw_input('Please Enter your Email: ')
password=raw_input('Please Enter Password: ')
print("\n******Don't Interrupt the Script******")
print('#script is Runing............\n')
chrome_options = webdriver.ChromeOptions() #going to chrome options
chrome_options.add_argument("--start-maximized")
prefs = {"profile.default_content_setting_values.notifications" : 2 #turn off all notifications
,"profile.managed_default_content_settings.images": 2} #disable images
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(chrome_options=chrome_options) # passing paramaters to chrome
driver.get('https://accounts.google.com')
time.sleep(3)
#giving Email-------------------
email = driver.find_element_by_id('Email')
email.send_keys(Email, Keys.RETURN)
#giving password----------------
time.sleep(3)
email = driver.find_element_by_id('Passwd')
email.send_keys(password, Keys.RETURN)
#credentials + attach with googleSheet------------------------------
scope = ['https://spreadsheets.google.com/feeds']
credentials = ServiceAccountCredentials.from_json_keyfile_name('stephens-31d8490b5bd2.json', scope)
google_sheet = gspread.authorize(credentials)
workSheet = google_sheet.open("Video Access Master Sheet").worksheet("Sheet1")
while True:
#fetch Records from Rows 2 to 50 and save on list-----------------
for i in range(2,51):
li_url=[]
li_email=[]
row=workSheet.row_values(i)
for b in row:
if 'youtu' in b:
li_url.append(b)
#find record which you append on list and then delete from googleSheet--------------------
cell = workSheet.find(b)
row = cell.row
col = cell.col
workSheet.update_cell(row,col, '')
print 'Fetching Values From Row '+str(i)+'....'
elif '#' in b:
li_email.append(b)
elif b=='':
continue
else:
continue
#*********************************************************
#getting length list of li_url and apply condition on it-----------------------------------------------
length=len(li_url)
if length==0:
continue
else:
try:
#getting URLs from list and put into driver.get---------------------------------------------------------
for a in li_url:
driver.get(a)
time.sleep(3)
driver.find_element_by_css_selector('.yt-uix-button-icon.yt-uix-button-icon-info.yt-sprite').click()
time.sleep(3)
driver.find_element_by_css_selector('.yt-uix-button.yt-uix-button-size-default.yt-uix-button-default.metadata-share-button').click()
time.sleep(2)
put_email=driver.find_element_by_css_selector('.yt-uix-form-input-textarea.metadata-share-contacts')
#getting emails from email list--------------------------------------------------------------
put_email.send_keys(li_email[0])
time.sleep(2)
driver.find_element_by_css_selector('.yt-uix-button.yt-uix-button-size-default.yt-uix-button-primary.sharing-dialog-button.sharing-dialog-ok').click()
time.sleep(4)
driver.find_element_by_xpath('.//*[#id="video-header"]/div/button[2]/span').click()
time.sleep(10)
#for notifications and alters--------------------------------------------
try:
driver.switch_to.alert.accept()
except NoAlertPresentException:
pass
except UnexpectedAlertPresentException:
pass
except:
traceback.print_exc
pass
print 'Row '+str(i)+' Successfully Updated. \n'
time.sleep(120) #while loop sleep for 20minuts
This is the error I got:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "<string>", line 56, in parse
File "<string>", line 35, in parse
cElementTree.ParseError: no element found: line 1, column 0

For some reason cell = workSheet.find(b) fails. Could be bad data in there; without seeing the input it's anyone's guess.
Since you already know the row number, you can avoid using cell = workSheet.find(b) by simply keeping track of the columns you're searching through and finally calling workSheet.update_cell(i, col, '') after copying the data.

python noob having troubles sending google voice text

I'm writing a simple little script to send me a text message when the Ultra Music Festival early bird tickets go on sale so I can snatch them up. When I came to writing this I figured python would be a quick way to achieve my goal. What I do is collect the links and then count them and determine if there is a change and send a google voice text message to a couple numbers. Here is my code ran against stackoverflow.
from googlevoice import Voice
from googlevoice.util import input
from bs4 import BeautifulSoup, SoupStrainer
from time import sleep
import urllib2
from array import *
#define login details
email = 'example#gmail.com'
password = 'password'
url = 'http://stackoverflow.com/questions'
def send_message(var_text):
voice = Voice()
voice.login(email, password)
phoneNumber = array('L',[9998675309, 9998675309])
for i in phoneNumber:
voice.send_sms(i, var_text)
#init
soup = BeautifulSoup(urllib2.urlopen(url).read(), parse_only=SoupStrainer('a'))
link_count = len(soup)
#start the loop
var = 1
while var == 1 : # This constructs an infinite loop
soup = BeautifulSoup(urllib2.urlopen(url).read(), parse_only=SoupStrainer('a'))
if link_count != len(soup):
string = str('Link Count Changed\n\nSite:\n' + url + '\nPrev:\n' + str(link_count) + '\nNew:\n' + str(len(soup)))
send_message(string)
print (string)
link_count = len(soup)
sleep(10)
pass
else:
print('Number of links ('+ str(link_count) + ') has not changed, going to sleep now.')
sleep(10)
pass
print "Good bye!"
Here is the error I keep getting (only seems to happen when sending to more then one number)
doesn't work array('L',[9998675309, 9998675309])
works array('L',[9998675309])
ERROR:
bash-3.2# python gvsendalert.py
Number of links (195) has not changed, going to sleep now.
Traceback (most recent call last):
File "gvsendalert.py", line 32, in <module>
send_message(string)
File "gvsendalert.py", line 19, in send_message
voice.send_sms(i, var_text)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googlevoice/voice.py", line 151, in send_sms
self.__validate_special_page('sms', {'phoneNumber': phoneNumber, 'text': text})
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googlevoice/voice.py", line 225, in __validate_special_page
load_and_validate(self.__do_special_page(page, data))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googlevoice/util.py", line 65, in load_and_validate
validate_response(loads(response.read()))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googlevoice/util.py", line 59, in validate_response
raise ValidationError('There was a problem with GV: %s' % response)
googlevoice.util.ValidationError: There was a problem with GV: {u'data': {u'code': 58}, u'ok': False}
Ok I've taken into consideration what some of you have posted and come out with this. For the number array sending my google voice number twice it will send 2 messages. If I put my friends number as the second it breaks it. Could this be because my friends number is not a google voice number? I have been able to send messages to this number using Google Voice and some other 3rd party iPhone applications so I would think the python module would work the same way.
Here is my 2nd Revision Code:
def send_message(var_text):
voice = Voice()
voice.login(email, password)
phoneNumber = ['myrealgooglenumber', 'myfriendsactualphonenumber']
for i in phoneNumber:
print( str('sending to: ') + str(i))
voice.send_sms(str(i), str(var_text))
sleep(5)
#init
soup = BeautifulSoup(urllib2.urlopen(url).read(), parse_only=SoupStrainer('a'))
link_count = len(soup)
#start the loop
var = 1
while var == 1 : # This constructs an infinite loop
soup = BeautifulSoup(urllib2.urlopen(url).read(), parse_only=SoupStrainer('a'))
if link_count != len(soup):
string = ('Link Count Changed\n\nSite:\n{0}\nPrev:\n{1}\nNew:\n{2}').format(url, link_count, len(soup))
send_message(string)
link_count = len(soup)
print (string)
sleep(10)
pass
else:
string = ('Number of links ({0}) has not changed, going to sleep now.').format(str(link_count))
print(string)
sleep(10)
pass
print "Good bye!"
Have tested with 2 google voice numbers and it works. Still doesn't work with non google voice numbers.

It looks like you're using ints for the phone numbers.
Phone numbers are not true numbers.
Try strings instead:
phoneNumber = ['9998675309', '9998675309']
Also, on a style note, have a look at string formatting:
string = 'Link Count Changed\n\nSite:\n{0}\nPrev:\n{1}\nNew:\n{2}').format(url, link_count, len(soup))

Google may have a timer to prevent you sending too many SMS messages back to back.
Perhaps you could try changing your loop to something like:
for i in phoneNumber:
voice.send_sms(i, var_text)
sleep(5)
One other thought, does it work better if you use 2 different phone numbers?