We Keep Coding

Using pytube on PyCharm

I have a script, which has been working fine in April 2021 when I created it, but now it gives me the following error. I'm not very experienced in coding, so if anyone can help me it would be great.
What I'm trying to do is simply download a song from youtube as a mp4. I can see that the error says there is something wrong with the pytube module imported, but I am not skilled enough to see what it is.
I'm using MacOS 12.1, Pycharm 2020.3, and Python 3.9.
Script:
import pytube
url = str('https://www.youtube.com/watch?v=gJLIiF15wjQ')
youtube = pytube.YouTube(url)
video = youtube.streams.get_by_itag(140)
video.download(output_path='/Users/clarajacobsen/Documents/TrueFIR/Klub100/Songs/', filename='test')
Error:
Traceback (most recent call last):
File "/Users/user/Documents/Folder1/venv/test.py", line 8, in <module>
video = youtube.streams.get_by_itag(140)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/__main__.py", line 292, in streams
return StreamQuery(self.fmt_streams)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/__main__.py", line 177, in fmt_streams
extract.apply_signature(stream_manifest, self.vid_info, self.js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/extract.py", line 409, in apply_signature
cipher = Cipher(js=js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/cipher.py", line 43, in __init__
self.throttling_plan = get_throttling_plan(js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/cipher.py", line 387, in get_throttling_plan
raw_code = get_throttling_function_code(js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/cipher.py", line 301, in get_throttling_function_code
code_lines_list = find_object_from_startpoint(js, match.span()[1]).split('\n')
AttributeError: 'NoneType' object has no attribute 'span'
After trying out solution 1, suggested by Sarim, error in PyCharm:
Traceback (most recent call last):
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/__main__.py", line 177, in fmt_streams
extract.apply_signature(stream_manifest, self.vid_info, self.js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/extract.py", line 409, in apply_signature
cipher = Cipher(js=js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/cipher.py", line 29, in __init__
self.transform_plan: List[str] = get_transform_plan(js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/cipher.py", line 197, in get_transform_plan
return regex_search(pattern, js, group=1).split(";")
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/helpers.py", line 129, in regex_search
raise RegexMatchError(caller="regex_search", pattern=pattern)
pytube.exceptions.RegexMatchError: regex_search: could not find match for iha=function\(\w\){[a-z=\.\(\"\)]*;(.*);(?:.+)}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/user/Documents/Folder1/venv/test.py", line 5, in <module>
video = youtube.streams.get_by_itag(140)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/__main__.py", line 292, in streams
return StreamQuery(self.fmt_streams)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/__main__.py", line 184, in fmt_streams
extract.apply_signature(stream_manifest, self.vid_info, self.js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/extract.py", line 409, in apply_signature
cipher = Cipher(js=js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/cipher.py", line 29, in __init__
self.transform_plan: List[str] = get_transform_plan(js)
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/cipher.py", line 197, in get_transform_plan
return regex_search(pattern, js, group=1).split(";")
File "/Users/user/Documents/Folder1/venv/lib/python3.9/site-packages/pytube/helpers.py", line 129, in regex_search
raise RegexMatchError(caller="regex_search", pattern=pattern)
pytube.exceptions.RegexMatchError: regex_search: could not find match for iha=function\(\w\){[a-z=\.\(\"\)]*;(.*);(?:.+)}
After trying to run it in Google Colab:
/usr/local/lib/python3.7/dist-packages/pytube/cipher.py in get_throttling_function_code(js)
299
300 # Extract the code within curly braces for the function itself, and merge any split lines
--> 301 code_lines_list = find_object_from_startpoint(js, match.span()[1]).split('\n')
302 joined_lines = "".join(code_lines_list)
303
AttributeError: 'NoneType' object has no attribute 'span'

To fix this issue, This doesnt depends on which operating system you are on or which python you are using. Follow these steps:
I used Colab for this, if you are using Google colab use it and test it.
Install Pytube with !pip install pytube
After installing pytube just shutdown the kernel and the application you are using for it. either VSCode, Jupyter notebook or Colab. shut down its kernel.
Then run the enviroment again and try importing and running your code.
It should run now.
or if it gives you the same error as before:
Go to the files where pytube is install and go to folder in pytube named "pytube" then go into "cipher.py" and open it.
Search for the line: 293. Where name = re.escape(get_throttling_function_name(js))
Replace name = "iha"
Then close all kernels and file you are running the code on. and restart them completely after shutting down.
These two solutions should work 100%. Solution that worked for me is first one.

As the error tells us, you have a NoneType object called youtube in line 8 which was created before in line 7. Did you check if the YouTube link or anything on that video page that concerns you has changed?

SyntaxError using gdata-python to get worksheets feed

I'm getting an occasional error when trying to fetch a list of worksheets from gdata. This does not happen for all spreadsheets, but will consistently happen to the same spreadsheet for a period of several days to weeks. I suspected permissions, but was unable to find any special permissions for the spreadsheets that cause the error. I'm using OAuth2, gdata 2.0.18, and Python 2.6.8.
Traceback (most recent call last):
File "/mnt/shared_from_host/snake/base/fetchers/google_spreadsheet/common.py", line 176, in get_worksheet_list
feed = client.get_worksheets(spreadsheet_id)
File "/home/ubuntu/.virtualenvs/snakeenv/lib/python2.6/site-packages/gdata/spreadsheets/client.py", line 108, in get_worksheets
**kwargs)
File "/home/ubuntu/.virtualenvs/snakeenv/lib/python2.6/site-packages/gdata/client.py", line 640, in get_feed
**kwargs)
File "/home/ubuntu/.virtualenvs/snakeenv/lib/python2.6/site-packages/gdata/client.py", line 278, in request
version=get_xml_version(self.api_version))
File "/home/ubuntu/.virtualenvs/snakeenv/lib/python2.6/site-packages/atom/core.py", line 520, in parse
tree = ElementTree.fromstring(xml_string)
File "<string>", line 86, in XML
SyntaxError: no element found: line 1, column 0
This seems to be from the request getting an empty string as the response.
Does anybody have any idea on why this might not work, or troubleshooting ideas? Thanks.

AttributeError: 'Response' object has no attribute '_dom'

I'm testing ebaysdk Python library that lets you connect to ebay. Now I'm trying examples from: https://github.com/timotheus/ebaysdk-python/
So far I got stuck at this example:
from ebaysdk.shopping import Connection as Shopping
shopping = Shopping(domain="svcs.sandbox.ebay.com", config_file="ebay.yaml")
response = shopping.execute('FindPopularItems',
{'QueryKeywords': 'Python'})
print response.disct()
When I run it. It gives me this error:
Traceback (most recent call last):
File "ebay-test.py", line 13, in <module>
{'QueryKeywords': 'Python'})
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/connection.py", line 123, in execute
self.error_check()
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/connection.py", line 193, in error_check
estr = self.error()
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/connection.py", line 305, in error
error_array.extend(self._get_resp_body_errors())
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/shopping/__init__.py", line 188, in _get_resp_body_errors
dom = self.response.dom()
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/response.py", line 229, in dom
return self._dom
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/response.py", line 216, in __getattr__
return getattr(self._obj, name)
AttributeError: 'Response' object has no attribute '_dom'
Am I missing something here or it could be some kind of bug in library?

Do you have a config file? I had a lot of problems getting started with this SDK. To get the yaml config file to work, I had to specify the directory that it was in. So in your example, it would be:
shopping = Shopping(domain="svcs.sandbox.ebay.com", config_file=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'ebay.yaml'))
You should also be able to specify debug=true in your Shopping() declaration as in Shopping(debug=True).
Make sure if you have not, to specify your APP_ID and other necessary values in the config file.

You have the wrong domain, it should be open.api.sandbox.ebay.com. See this page on the ebaysdk github.

AttributeError: 'module' object has no attribute 'loads' in TwitterAPI call (iterating over tweets)?

When running the sample code on TwitterAPI page here , I receive the following error.
Traceback (most recent call last):
File "D:/Cloud/Dropbox/Coding/Python/EUR - BigData & Analytics/Workshops/Tutorial 2/twitter_query_scraper_revisited.py", line 76, in <module>
for tweet in tweet_sequence:
File "C:\Python33\lib\site-packages\TwitterAPI\TwitterAPI.py", line 116, in __iter__
for item in self.get_iterator():
File "C:\Python33\lib\site-packages\TwitterAPI\TwitterAPI.py", line 113, in get_iterator
return RestIterator(self.response)
File "C:\Python33\lib\site-packages\TwitterAPI\TwitterAPI.py", line 139, in __init__
resp = response.json()
File "C:\Python33\lib\site-packages\requests\models.py", line 741, in json
return json.loads(self.text, **kwargs)
AttributeError: 'module' object has no attribute 'loads'
I tried running through the various code, but I've been unable to solve it.
This is the code I'm running:
from TwitterAPI import TwitterAPI
api = TwitterAPI(consumer_key, consumer_secret, access_token_key, access_token_secret)
r = api.request('search/tweets', {'q':'pizza'})
for item in r.get_iterator():
print item
I'm running the latest version of TwitterAPI (2.1.13), requests 2.2.1 and am running Python 3.3.4 on Windows 8.
Any help is greatly appreciated.

You have a json module that masks the standard library module, and it is imported instead.
Since your own json module has no loads() function, the request library trips up.
You can find what file this is by running:
import json
print(json.__file__)
Rename that module to something else.

Running mapreduce jobs on google app engine

I'm running the demo that comes with the mapreduce framework. It's giving me an error:
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/path/to/mapreduce/base_handler.py", line 68, in post
self.handle()
File "/path/to/mapreduce/handlers.py", line 431, in handle
self.aggregate_state(state, shard_states)
File "/path/to/mapreduce/handlers.py", line 462, in aggregate_state
context.COUNTER_MAPPER_CALLS))
File "/path/to/mapreduce/model.py", line 257, in get
return self.counters.get(counter_name, 0)
AttributeError: 'list' object has no attribute 'get'
Is this something I'm doing wrong, does the demo not work? Is there more updated code somewhere else?
This is using the code from http://appengine-mapreduce.googlecode.com/svn/trunk/

Not familiar with that code, but the latest code is the MapReduce Bundle you can download from the SDK:
https://developers.google.com/appengine/downloads
It comes with a bit of a demo. I was able to follow this and get this to work:
http://code.google.com/p/appengine-mapreduce/wiki/GettingStartedInPython
Here's some additional notes I had when I was trying to get MapReduce running.
http://eatdev.tumblr.com/post/17983355135/using-mapreduce-with-django-nonrel-on-app-engine