We Keep Coding

Stuck with pytube

Oh my python guys, please help me.
One of my python projects suddenly stopped working for no reason.
I'm using pytube module and when i try to run the code i get this error:
Traceback (most recent call last):
File "C:\Users\giova\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\contrib\search.py", line 94, in fetch_and_parse
sections = raw_results['contents']['twoColumnSearchResultsRenderer'][
KeyError: 'twoColumnSearchResultsRenderer' fetch_and_parse
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\giova\OneDrive\Desktop\Coding\Python\youtubeapp.py", line 38, in <module>
videoSearch()
File "C:\Users\giova\OneDrive\Desktop\Coding\Python\youtubeapp.py", line 21, in videoSearch
availableResults = len(vid.results)
File "C:\Users\giova\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\contrib\search.py", line 62, in results
videos, continuation = self.fetch_and_parse() results
File "C:\Users\giova\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\contrib\search.py", line 97, in fetch_and_parse fetch_and_parse
sections = raw_results['onResponseReceivedCommands'][0][
KeyError: 'onResponseReceivedCommands'
This is not even the only error i get, sometimes i got "http error 410: gone" error or some like this. I haven't changed the code for about two weeks (it was working two weeks ago) and it stopped working. I don't know what is happening to my code.
This is the full code:
from pytube import Search, YouTube
print("================================\n What do you want to do?: ")
availableChoose = [
'1 Search videos',
'...',
'================================'
]
for choose in availableChoose:
print(choose)
userChoose = input()
userChoose = userChoose.lower()
def videoSearch():
userSearch = input("Enter the title of the video you want to search: ")
vid = Search(userSearch)
availableResults = len(vid.results)
strAvailableResults = str(availableResults)
print("The available results are " + strAvailableResults)
vidResultsList = vid.results
vidResultsList = str(vidResultsList)
vidResultsList = vidResultsList.replace("<pytube.__main__.YouTube object: videoId=", "")
vidResultsList = vidResultsList.replace(">", "")
vidResultsList = vidResultsList.replace("[", "")
vidResultsList = vidResultsList.replace("]", "")
vidResultsList = vidResultsList.replace(" ", "")
vidResultsList = vidResultsList.split(',')
for vidResultsObject in vidResultsList:
vidLink = ("https://www.youtube.com/watch?v=" + vidResultsObject)
vidTempObject = YouTube(vidLink)
print(vidTempObject.title + " - " + vidLink)
if(userChoose == "search" or userChoose == "search video" or userChoose == "search videos" or userChoose == "1"):
videoSearch()

pytube's 11.0.0 API docs list the Search.fetch_and_parse() method.
The corresponding source-code shows that it internally handles a KeyError for onResponseReceivedCommands:
# Initial result is handled by try block, continuations by except block
try:
sections = raw_results['contents']['twoColumnSearchResultsRenderer'][
'primaryContents']['sectionListRenderer']['contents']
except KeyError:
sections = raw_results['onResponseReceivedCommands'][0][
'appendContinuationItemsAction']['continuationItems']
This method is an inner one, that is used by your vid.results call.
Might be, that the Youtube API has changed there response and your version of pytube is not fitting anymore.
Bug already filed
See pytube issue #1082 and issue #1106.
Meanwhile use another branch
tfdahlin's fork has a bugfixed version. It was already proposed as Pull-Request to the project:
PR #1090, opened on 2021-08-13 (currently waiting for approval).

I am also struggling to get around the search results provided by pytube Search module. Since search results are looking like objects, I was thinking (lazily) that I cannot convert the object list in to strings.
After modifying your function as below, the results are printed as youtube links.
'''
def videoSearch(search_list): #provide the search.results list as input
for result in search_list: #Begin video id extraction
result = str(result)
temp = result.replace("<pytube.__main__.YouTube object: videoId=", "")
temp = temp.split(',')[0] #get the correct video id
vidLink = ("https://www.youtube.com/watch?v=" + temp)
print(vidLink)
'''
Try it out.

Python scripting with ete3 to query NCBI's Taxonomy: "sqlite3 Warning (can only execute one statement at a time)"

I am using this script:
import csv
import time
import sys
from ete3 import NCBITaxa
ncbi = NCBITaxa()
def get_desired_ranks(taxid, desired_ranks):
lineage = ncbi.get_lineage(taxid)
names = ncbi.get_taxid_translator(lineage)
lineage2ranks = ncbi.get_rank(names)
ranks2lineage = dict((rank,taxid) for (taxid, rank) in lineage2ranks.items())
return{'{}_id'.format(rank): ranks2lineage.get(rank, '<not present>') for rank in desired_ranks}
if __name__ == '__main__':
file = open(sys.argv[1], "r")
taxids = []
contigs = []
for line in file:
line = line.split("\n")[0]
taxids.append(line.split(",")[0])
contigs.append(line.split(",")[1])
desired_ranks = ['superkingdom', 'phylum']
results = list()
for taxid in taxids:
results.append(list())
results[-1].append(str(taxid))
ranks = get_desired_ranks(taxid, desired_ranks)
for key, rank in ranks.items():
if rank != '<not present>':
results[-1].append(list(ncbi.get_taxid_translator([rank]).values())[0])
else:
results[-1].append(rank)
i = 0
for result in results:
print(contigs[i] + ','),
print(','.join(result))
i += 1
file.close()
The script takes taxids from a file and fetches their respective lineages from a local copy of NCBI's Taxonomy database. Strangely, this script works fine when I run it on small sets of taxids (~70, ~100), but most of my datasets are upwards of 280k taxids and these break the script.
I get this complete error:
Traceback (most recent call last):
File "/data1/lstout/blast/scripts/getLineageByETE3.py", line 31, in <module>
ranks = get_desired_ranks(taxid, desired_ranks)
File "/data1/lstout/blast/scripts/getLineageByETE3.py", line 11, in get_desired_ranks
lineage = ncbi.get_lineage(taxid)
File "/data1/lstout/.local/lib/python2.7/site-packages/ete3/ncbi_taxonomy/ncbiquery.py", line 227, in get_lineage
result = self.db.execute('SELECT track FROM species WHERE taxid=%s' %taxid)
sqlite3.Warning: You can only execute one statement at a time.
The first two files from the traceback are simply the script I referenced above, the third file is one of ete3's. And as I stated, the script works fine with small datasets.
What I have tried:
Importing the time module and sleeping for a few milliseconds/hundredths of a second before/after my offending lines of code on lines 11 and 31. No effect.
Went to line 227 in ete3's code...
result = self.db.execute('SELECT track FROM species WHERE taxid=%s' %merged_conversion[taxid])
and changed the "execute" function to "executescript" in order to be able to handle multiple queries at once (as that seems to be the problem). This produced a new error and led to a rabbit hole of me changing minor things in their script trying to fudge this to work. No result. This is the complete offending function:
def get_lineage(self, taxid):
"""Given a valid taxid number, return its corresponding lineage track as a
hierarchically sorted list of parent taxids.
"""
if not taxid:
return None
result = self.db.execute('SELECT track FROM species WHERE taxid=%s' %taxid)
raw_track = result.fetchone()
if not raw_track:
#perhaps is an obsolete taxid
_, merged_conversion = self._translate_merged([taxid])
if taxid in merged_conversion:
result = self.db.execute('SELECT track FROM species WHERE taxid=%s' %merged_conversion[taxid])
raw_track = result.fetchone()
# if not raise error
if not raw_track:
#raw_track = ["1"]
raise ValueError("%s taxid not found" %taxid)
else:
warnings.warn("taxid %s was translated into %s" %(taxid, merged_conversion[taxid]))
track = list(map(int, raw_track[0].split(",")))
return list(reversed(track))
What bothers me so much is that this works on small amounts of data! I'm running these scripts from my school's high performance computer and have tried running on their head node and in an interactive moab scheduler. Nothing has helped.

bibtexparser - pyparsing.ParseException: Expected end of text

I'm using bibtexparser to parse a bibtex file.
import bibtexparser
with open('MetaGJK12842.bib','r') as bibfile:
bibdata = bibtexparser.load(bibfile)
While parsing I get the error message:
Could not parse properly, starting at
#article{Frenn:EvidenceBasedNursing:1999,
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pyparsing.py", line 3183, in parseImpl
raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected end of text (at char 5773750),
(line:47478, col:1)`
The line refers to the following bibtex entry:
#article{Frenn:EvidenceBasedNursing:1999,
author = {Frenn, M.},
title = {A Mediterranean type diet reduced all cause and cardiac mortality after a first myocardial infarction [commentary on de Lorgeril M, Salen P, Martin JL, et al. Mediterranean dietary pattern in a randomized trial: prolonged survival and possible reduced cancer rate. ARCH INTERN MED 1998;158:1181-7]},
journal = {Evidence Based Nursing},
uuid = {15A66A61-0343-475A-8700-F311B08BB2BC},
volume = {2},
number = {2},
pages = {48-48},
address = {College of Nursing, Marquette University, Milwaukee, WI},
year = {1999},
ISSN = {1367-6539},
url = {},
keywords = {Treatment Outcomes;Mediterranean Diet;Mortality;France;Neoplasms -- Prevention and Control;Phase One Excluded - No Assessment of Vegetable as DV;Female;Phase One - Reviewed by Hao;Myocardial Infarction -- Diet Therapy;Diet, Fat-Restricted;Phase One Excluded - No Fruit or Vegetable Study;Phase One Excluded - No Assessment of Fruit as DV;Male;Clinical Trials},
tags = {Phase One Excluded - No Assessment of Vegetable as DV;Phase One Excluded - No Fruit or Vegetable Study;Phase One - Reviewed by Hao;Phase One Excluded - No Assessment of Fruit as DV},
accession_num = {2000008864. Language: English. Entry Date: 20000201. Revision Date: 20130524. Publication Type: journal article},
remote_database_name = {rzh},
source_app = {EndNote},
EndNote_reference_number = {4413},
Secondary_title = {Evidence Based Nursing},
Citation_identifier = {Frenn 1999a},
remote_database_provider = {EBSCOhost},
publicationStatus = {Unknown},
abstract = {Question: text.},
notes = {(0) abstract; commentary. Journal Subset: Core Nursing; Europe; Nursing; Peer Reviewed; UK \& Ireland. No. of Refs: 1 ref. NLM UID: 9815947.}
}
What is wrong with this entry?

It seems that the issue has been addressed and resolved in the project repository (see Issue 147)
Until the next release, installing the library from the git repository can serve as a temporary fix.
pip install --upgrade git+https://github.com/sciunto-org/python-bibtexparser.git#master

I had this same error and found an entry near the line mentioned in the error that had a line like this
...
year = {1959},
month =
}
When I removed the null month item it parsed for me.

NameError: name '<name>' is not defined, when using % format syntax for strings

Traceback (most recent call last):
File "C:\Users\Simon\Downloads\rpgbs.py", line 72, in <module>
print("%(chara) has %(health) HP." % {chara:names[k], health:str(health[k])})
NameError: name 'chara' is not defined
I get the above error when running this code. The relevant sections are below.
When I run print(names["Enemy"], health["Enemy"]) on its own, that seems to work just fine. But when I use formatting syntax (from my understanding print("%(author) likes spam.") % {author:"Simon"}), it goes right down the drain. Does it have anything to do with the fact that I'm trying to iterate it?
names = {'Player' : str(input("What's your name, hero?: ")),
'Enemy' : str(input("And who will you be duelling with?"))}
if names["Enemy"]=="UNDERTALE":
names["Enemy"]=namechoose(["Froggit", "Sans", "Flowey", "Chara", "ASRIEL DREEMUR", "Toriel", "Undyne", "Mettaton"],1)
elif names["Enemy"]=="POKEMON":
names["Enemy"]=namechoose(["Charizard","Blastoise", "Venusaur", "Arceus", "Dialga", "Palkia", "Groudon"],1)
elif names["Enemy"]=="DRAGON QUEST":
names["Enemy"]=namechoose(["Corvus", "Aquila", "Hootingham-Gore", "Goresby-Purrvis", "Stella", "The Almighty", "Slime", "Metal Slime"],1)
startinghealth=int(input("How much health do you wish to start with? 150 is recommended."))
health = {'Player' : startinghealth,
'Enemy' : startinghealth}
while health["Enemy"]>0 and health["Player"]>0:
for k in names:
print(names[k], health[k])
print("%(chara) has %(health) HP." % {chara:names[k], health:str(health[k])})

You missed s character after the brace, also you use chara as variable in this line not as string key for hash.
That why you have that error that say that you don't have chara variable defined. Correct line should look like this:
print("%(chara)s has %(health)s HP." % {'chara':names[k], 'health':str(health[k])})

Weird TypeError in jython/python?

I'm bukkit jython/python plugin programmer. Last few days, I'm struggling with this problem. I have to add an potion effect to an user, it's not problem if I enter effect name, duration and amplifier manually in code, but when I want to get them from config, I get this error:
13:38:20 [SEVERE] Could not pass event PlayerInteractEvent to ItemEffect v1.0
Traceback (most recent call last):
File "<iostream>", line 126, in onPlayerInteractEvent
TypeError: addPotionEffect(): 1st arg can't be coerced to org.bukkit.potion.Poti
onEffect
Here's that part of code:
effectname = section.getString("%s.effect"%currentKey)
duration = section.getInt("%s.duration"%currentKey)
durationinticks = duration * 20
geteffectname = "PotionEffectType.%s"%effectname
getpotioneffect = "PotionEffect(%s, %i, 1)"%(geteffectname, durationinticks)
geteffectname = "PotionEffectType.%s"%effectname
if iteminhand == currentKey:
event.getPlayer().addPotionEffect(getpotioneffect)
When I print getpotioneffect out, I get:
13:38:20 [INFO] PotionEffect(PotionEffectType.SPEED, 600, 1)
which is okay, and should work. I tested it without getting informations from config, and it works perfectly... To sum up, code above is not working, but below one works:
getpotioneffect = PotionEffect(PotionEffectType.SPEED, 600, 1)
if iteminhand == currentKey:
event.getPlayer().addPotionEffect(getpotioneffect)
Link to javadocs of this event!
http://jd.bukkit.org/rb/apidocs/org/bukkit/entity/LivingEntity.html#addPotionEffect(org.bukkit.potion.PotionEffect)
Thanks!

In your first snippet, getpotioneffect is a string. You can check it adding print type(getpotioneffect) somewhere.
What you want is to replace this :
geteffectname = "PotionEffectType.%s"%effectname
getpotioneffect = "PotionEffect(%s, %i, 1)"%(geteffectname, durationinticks)
with this:
effect_type = getattr(PotionEffectType, effectname)
potion_effect = PotionEffect(effect_type, durationinticks, 1)