Python Web Scraping (edX) - python

I am trying to download a course on edx. Please refer to the following readme (https://github.com/coursera-dl/edx-dl/blob/master/README.md). I installed Anaconda for Windows (I am running Windows 10 in Parallel Desktop).
In the last step, I typed in the following command: C:\edx-dl-master\edx-dl-master>edx-dl -u user#user.com COURSE-URL and upon entering my edX password, get the following:
Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\runpy.py", line 193, in
_run_module_as_main
"__main__", mod_spec)
File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\Scripts\edx-dl.exe\__main__.py", line 9, in <module>
File "c:\programdata\anaconda3\lib\site-packages\edx_dl\edx_dl.py", line 1011, in main
for selected_course in selected_courses}
File "c:\programdata\anaconda3\lib\site-packages\edx_dl\edx_dl.py", line 1011, in <dictcomp>
for selected_course in selected_courses}
File "c:\programdata\anaconda3\lib\site-packages\edx_dl\edx_dl.py", line 186, in get_available_sections
sections = page_extractor.extract_sections_from_html(page, BASE_URL)
File "c:\programdata\anaconda3\lib\site-packages\edx_dl\parsing.py", line 403, in extract_sections_from_html
for i, section_soup in enumerate(sections_soup, 1)]
File "c:\programdata\anaconda3\lib\site-packages\edx_dl\parsing.py", line 403, in <listcomp>
for i, section_soup in enumerate(sections_soup, 1)]
File "c:\programdata\anaconda3\lib\site-packages\edx_dl\parsing.py", line 392, in _make_subsections
for i, s in enumerate(subsections_soup, 1)]
File "c:\programdata\anaconda3\lib\site-packages\edx_dl\parsing.py", line 392, in <listcomp>
for i, s in enumerate(subsections_soup, 1)]
AttributeError: 'NoneType' object has no attribute 'string'
I am new to Python, and am not sure what remedial measures I can take.

Try
git clone https://github.com/coursera-dl/edx-dl/blob/master/README.md
Install git first.

Related

How to solve indexerror when running manim examples?

I just start learning manim and Im not so good at python.
when I was running manim examples given by the manim kindergarten, there was a error about index but I really don't know how to solve it.
As follows
manimgl example_scenes.py OpeningManimExample
Here is the information
Traceback (most recent call last):
File "D:\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\Python310\Scripts\manimgl.exe\__main__.py", line 7, in <module>
File "C:\Users\10447\Desktop\manim\manimlib\__main__.py", line 25, in main
scene.run()
File "C:\Users\10447\Desktop\manim\manimlib\scene\scene.py", line 148, in run
self.construct()
File "C:\Users\10447\Desktop\manim\.\example_scenes.py", line 14, in construct
intro_words = Text("""
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\text_mobject.py", line 425, in __init__
super().__init__(
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\text_mobject.py", line 130, in __init__
super().__init__(text, height=height, **kwargs)
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\string_mobject.py", line 68, in __init__
super().__init__(
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 76, in __init__
self.init_svg_mobject()
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 104, in init_svg_mobject
submobs = self.mobjects_from_file(self.get_file_path())
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\string_mobject.py", line 115, in mobjects_from_file
submobs = super().mobjects_from_file(file_path)
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 132, in mobjects_from_file
return self.mobjects_from_svg(svg)
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 187, in mobjects_from_svg
mob = self.path_to_mobject(shape)
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 240, in path_to_mobject
return VMobjectFromSVGPath(path, **self.path_string_config)
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 306, in __init__
super().__init__(**kwargs)
File "C:\Users\10447\Desktop\manim\manimlib\mobject\types\vectorized_mobject.py", line 113, in __init__
super().__init__(**kwargs)
File "C:\Users\10447\Desktop\manim\manimlib\mobject\mobject.py", line 110, in __init__
self.init_points()
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 314, in init_points
self.handle_commands()
File "C:\Users\10447\Desktop\manim\manimlib\mobject\svg\svg_mobject.py", line 352, in handle_commands
if self.has_new_path_started():
File "C:\Users\10447\Desktop\manim\manimlib\mobject\types\vectorized_mobject.py", line 502, in has_new_path_started
return self.consider_points_equal(points[-3], points[-2])
IndexError: index -3 is out of bounds for axis 0 with size 0
It seems that the problem is about vectorized_mobject.py.
I've search all google but it seems no one has the same problem like mine.
In the docs of manim kindergarten I found no solution.
Hopefully some here once had the same problem and please let me know how to solve it.
Really thanks if someone could help me.
If you want to use Manim. I'd suggest you install ManimCE version from Manim Community. As it is more stable.
If you want to use, Manimgl to reproduce the animations from 3b1b videos.
Make sure you have FFmpeg and latex properly installed (Using
Miktex? make sure you enable it to install tex packages on it's own
or install all the latex packages).
Create a python virtual environment. Enable the venv you have just created.
Install manim using pip with pip install manimgl. Don't install it from
the source, as there are still some bugs at the time of writing.
If you followed all the steps correctly then running manimgl example_scenes.py OpeningManimExample should work.

Hi, I have some issue with the python pipreqs. It keeps telling me the usage instead of creating the right file

As you can see, I use the right command and it keeps telling me the usage instead of creating the right file for me.
enter image description here
(WebScraping) G:\>pipreqs G:\PY CODE\WebScraping
Usage:
pipreqs [options] [<path>]
And if I just randomly input the path file, it will tell me, file doesn`t exist.
(WebScraping) G:\>pipreqs s
Traceback (most recent call last):
File "D:\anaconda\envs\WebScraping\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\anaconda\envs\WebScraping\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\anaconda\envs\WebScraping\Scripts\pipreqs.exe\__main__.py", line 7, in <module>
File "D:\anaconda\envs\WebScraping\lib\site-packages\pipreqs\pipreqs.py", line 488, in main
init(args)
File "D:\anaconda\envs\WebScraping\lib\site-packages\pipreqs\pipreqs.py", line 478, in init
generate_requirements_file(path, imports, symbol)
File "D:\anaconda\envs\WebScraping\lib\site-packages\pipreqs\pipreqs.py", line 157, in generate_requirements_file
with _open(path, "w") as out_file:
File "D:\anaconda\envs\WebScraping\lib\contextlib.py", line 135, in __enter__
return next(self.gen)
File "D:\anaconda\envs\WebScraping\lib\site-packages\pipreqs\pipreqs.py", line 81, in _open
file = open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: 's\\requirements.txt'
I finally solve this issue when I cd my host path in cmd to the document path without giving arguments in pipreqs order like this:
(WebScraping) G:\PY CODE\WebScraping>pipreqs --encoding=utf8
INFO: Successfully saved requirements file in G:\PY CODE\WebScraping\requirements.txt
But how can I use the normal orders like eveyone else:
G:\PY CODE\WebScraping
This keeps bothering me
I wonder is there someth
Try adding quotes:
pipreqs "G:\PY CODE\WebScraping"

TypeError: must provide a stream to wrap

When I type pipenv shell in my command line to activate virtual environment I got this error:
Traceback (most recent call last):
File "c:\python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\asus\AppData\Roaming\Python\Python39\Scripts\pipenv.exe\__main_
_.py", line 4, in <module>
File "C:\Users\asus\AppData\Roaming\Python\Python39\site-packages\pipenv\__ini
t__.py", line 41, in <module>
stdout = get_text_stream("stdout")
File "C:\Users\asus\AppData\Roaming\Python\Python39\site-packages\pipenv\vendo
r\vistir\misc.py", line 1191, in get_text_stream
return get_wrapped_stream(sys_stream, encoding)
File "C:\Users\asus\AppData\Roaming\Python\Python39\site-packages\pipenv\vendo
r\vistir\misc.py", line 969, in get_wrapped_stream
raise TypeError("must provide a stream to wrap")
TypeError: must provide a stream to wrap
if you know how to overcome this error, please answer. Thank you.
I think what your problem is, that you're missing stdout. I've had a similar problem with a script that was invoked from KDE. It kind of failed with a similar error message.
My fix for the moment was to simply redirect the stdout to /dev/null. I assume that this won't quite work, considering you are working from within a shell that should have stdin/stout. You might have to reconsider your working environment. But just a guess.

flake8 internal error in regular expression engine

I'm trying to run flake8 on a docker django built like described here (tutorial page)
when building the docker image I get an error from flake8 which is run in an docker-compose file with like so
$ flake8 --ignore=E501,F401 .
multiprocessing.pool.RemoteTraceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 48, in
return list(map(*
File "/usr/local/lib/python3.8/site-packages/flake8/checker.py", line 666, in
return checker.run_checks()
File "/usr/local/lib/python3.8/site-packages/flake8/checker.py", line 598, in run_checks
self.run_ast_checks()
File "/usr/local/lib/python3.8/site-packages/flake8/checker.py", line 495, in run_ast_checks
checker = self.run_check(plugin, tree=ast)
File "/usr/local/lib/python3.8/site-packages/flake8/checker.py", line 426, in run_check
self.processor.keyword_arguments_for( File "/usr/local/lib/python3.8/site-packages/flake8/processor.py", line 241, in keyword_arguments_for arguments[param] = getattr(self, param)
File "/usr/local/lib/python3.8/site-packages/flake8/processor.py", line 119, in file_tokens
self._file_tokens = list(
File "/usr/local/lib/python3.8/tokenize.py", line 525, in _tokenize
pseudomatch = _compile(PseudoToken).match(line, pos)
RuntimeError: internal error in regular expression engine
The above exception was the direct cause of the following exception: Traceback (most recent call last):
File "/usr/local/bin/flake8", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/flake8/main/cli.py", line 18, in main
app.run(argv)
File "/usr/local/lib/python3.8/site-packages/flake8/main/application.py", line 393, in run
self._run(argv)
File "/usr/local/lib/python3.8/site-packages/flake8/main/application.py", line 381, in _run
self.run_checks()
File "/usr/local/lib/python3.8/site-packages/flake8/main/application.py", line 300, in run_checks
self.file_checker_manager.run()
File "/usr/local/lib/python3.8/site-packages/flake8/checker.py", line 329, in run
self.run_parallel()
File "/usr/local/lib/python3.8/site-packages/flake8/checker.py", line 293, in run_parallel
for ret in pool_map:
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 448, in <genexpr>
return (item for chunk in result for item in chunk)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 865, in next
raise value
RuntimeError: internal error in regular expression engine
When I run flake8 with the --verbose flag, I get an error like this:
Fatal Python error: deletion of interned string failed
Python runtime state: initialized
KeyError: 'FILENAME_RE'
from the tokenizer.py
Does anyone know how to solve this?
Additional Data:
running docker-compose v1.25.4 on an raspberry 3 with buster lite.
Installed and compiled Python 3.8.2 from source with the flag --enableloadable-sqlite
Thanks for helping!
If you don't care for flake8, then just delete that line. It will work. I had the same problem.
I know this isn't an answer. I would add a comment instead if I could.
I encountered a very similar error profile when I had refactored and left a lot of data which formerly had been in the top of the project down in what became the package folder. There were 33621 files including .venv, .nox, .pytest_cache, .coverage.
File "/usr/lib/python3.8/multiprocessing/pool.py", line 448, in <genexpr>
return (item for chunk in result for item in chunk)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
IndexError: string index out of range
If you see a similar signature (the parallel processing engine being overwhelmed in some way and throwing an exception), you may want to review your project directory structure and make sure that you did not put this kind of data in your sources directories.

Running install mod_dav_svn and can't decipher thread error

I am relatively new to Python, and at the same time, am attempting to install mod_dav_svn into my Apache web server. I am looking to get some idea of the scope of the error I'm receiving.
At the command line, I type in 'sudo yum install mod_dav_svn' and receive this output:
Loaded plugins: fastestmirror
Determining fastest mirrors
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in ?
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 229, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 104, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 339, in doCommands
self._getTs(needTsRemove)
File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 101, in _getTs
self._getTsInfo(remove_only)
File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 112, in _getTsInfo
pkgSack = self.pkgSack
File "/usr/lib/python2.4/site-packages/yum/init.py", line 591, in
pkgSack = property(fget=lambda self: self._getSacks(),
File "/usr/lib/python2.4/site-packages/yum/init.py", line 434, in _getSacks
self.repos.populateSack(which=repos)
File "/usr/lib/python2.4/site-packages/yum/repos.py", line 223, in populateSack
self.doSetup()
File "/usr/lib/python2.4/site-packages/yum/repos.py", line 71, in doSetup
self.ayum.plugins.run('postreposetup')
File "/usr/lib/python2.4/site-packages/yum/plugins.py", line 176, in run
func(conduitcls(self, self.base, conf, **kwargs))
File "/usr/lib/yum-plugins/fastestmirror.py", line 181, in postreposetup_hook
all_urls = FastestMirror(all_urls).get_mirrorlist()
File "/usr/lib/yum-plugins/fastestmirror.py", line 333, in get_mirrorlist
self._poll_mirrors()
File "/usr/lib/yum-plugins/fastestmirror.py", line 376, in _poll_mirrors
pollThread.start()
File "/usr/lib/python2.4/threading.py", line 416, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
The only other question I could find with a similar error was this one: https://stackoverflow.com/search?q=can%27t+start+new+thread+python, but I'm not sure it has to do with running too many threads since I am the only one using the server, and this is one of the first python commands I have used. Could someone point me in the right direction, or towards some material that may help me troubleshoot the issue? Thanks!
Disable yum-fastermirror and you should be able to complete the install.
yum --disableplugin=fastestmirror update
Are you on a virtual machine?

Categories