Python import BeautifulSoup not working - python

I have a Python script that wants to use BeautifulSoup:
from bs4 import BeautifulSoup
When I run the script from the command line, it works fine. When I run the script externally, from a browser, it dies on that line. The web logs say:
ImportError: No module named bs4
I've also tried, with the same result:
import bs4
import BeautifulSoup
I installed the module from a tarball, and it now resides in a folder that is in my Python path:
/usr/home/myName/.local/lib/python2.7/site-packages/
I have made sure the permissions for the bs4 directory, and all .py and .pyc files in the folder allow execution (chmod 775 *.py), and I've checked to see that both internally and externally the same version of Python is being run (2.7.9 (default, Jan 12 2015, 16:33:18)
[GCC 4.2.1 20070719 [FreeBSD]])

You have not installed BeautifulSoup for the benefit of all the users on your computer. Instead, you installed only for the benefit of the user named "myName". In particular, you haven't installed it for the use of the user that runs the web server (often named 'www' or 'www-data').
If you can, install BeautifulSoup in the system-wide location.
Otherwise, you can modify your script like so:
import sys
sys.path[0:0] = ['/usr/home/myName/.local/lib/python2.7/site-packages/']
from bs4 import BeautifulSoup

Thanks to #robᵩ (phi) for pointing out what was wrong: because this script was on a shared server, I was unable to give execute permission to the "user" running the web server (in this case, "nobody"). I was able to run the script using ssh, since I was the user in that case.
Because I was unable to convince the hosting service to install BeautifulSoup for all users, I ended up using a different library, lxml.html, which my host server does install with Python.
(I found lxml.html a pleasure to work with, as well)

Related

Import statements not working / I think I broke Python / file structure issues?

[I am new to Python (and programming in general) and will definitely say something stupid in this question.]
I had two python programs. In one of them the import statements were working. And in the other one the import statements were not working.
I suspected this had something to do with the file location of the modules relative to the Python files.
It turned out the program that wasn't working was in a sub folder of the program that was working.
So, as an experiment, I tried moving the venv folder into the sub folder where the other program was, but I ended up canceling that once I discovered that I would need to replace some of the files. (Due to the fact that is already had a venv folder.)
Then, as an experiment, I tried renaming the venv folder to "venv1" just to see if the good program would run. I was not surprised when it didn't.
But then I renamed it back to "venv," and it still wasn't working.
from bs4 import BeautifulSoup
import requests
import json, requests
import urllib.request
import bs4 as bs
import urllib
# .... etc ...
output
ModuleNotFoundError: No module named 'bs4'
...
...
...
oh, and if I try:
#from bs4 import BeautifulSoup
import requests
import json, requests
import urllib.request
import bs4 as bs
import urllib
# .... etc ...
Output:
ModuleNotFoundError: No module named 'requests'
I tried pip installing them again (my terminal doesn't recognize sudo pip install) and this is what I got
PS C:\Users\****\Desktop> pip install requests
Requirement already satisfied: requests in c:\users\****\appdata\local\programs\python\python310\lib\site-packages (2.27.1)
I thought maybe I'd look this one up, but the folder "appdata" doesn't exist on my computer, in that location.
What happened and how can I fix it?
The appdata folder should exist in that location. It is a hidden folder, and by default, Windows won't display hidden files/folders. You can view it by pressing WIN+R, and then typing "appdata", and clicking "OK". It should then come up in a file explorer window.
The python packages are installed, but not visible to the scripts. It sounds like you virtual environment may be incorrectly set up. If you open a CMD prompt, and then type in python -m site, it will show you the locations of your python's system path. You should see the install locations for the packages, in this case, you'll probably see the following: C:\\Users\\****\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages.

lxml not being recognized in bs4: Python 3 on mac

So, I have a mac on High Sierra and I am trying to import and use an api. This is api is a python3 api and uses bs4, and specifically is using lxml within bs4 in order to parse a webpage.
However, I am having an issue getting bs4 to recognize that I have lxml installed on my machine. I have installed both of them using pip, and both appear to have installed correctly. I can write a program with ‘import bs4’ and ‘import lxml’ at the top and it compiles and runs perfectly fine. However, no matter what I do I always get the following error when I run a program using this api.
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
On top of this, when I run the following code
import lxml
import bs4
print(bs4.builder.builder_registry.builders)
the output is
[<class 'bs4.builder._htmlparser.HTMLParserTreeBuilder'>]
With no lxml listed.
I have tried everything I have found on the various stack overflow threads related to this. I have uninstalled and reinstalled both lxml and bs4 through various methods(pip, easy install, manually installing, homebrew). I've manually linked lxml from brew. And other things ive probably forgotten. However I cant get it to work.
Anyone have any ideas/has anyone gone through this before. Its possible I'm completely missing something small or stupid, since I've never messed with bs4 before, but I dont know.
I'm not exactly sure what's the cause, but I had a similar case with a Flask app I'm working on. I worked around it by importing bs4 locally in the function where I needed it.
One symptom I had was when I logged bs4.builder.builder_registry.builders at the top of my module, the logs ended up showing two entries: first with the proper builders and then with only HTML.

Cannot use suds from VS with IronPython

I'm developing a WPF IronPython application with VS 2015. I have to send a SOAP call and to read the answer so I'm trying to get suds to work. Since both pip and easy_install failed to install it, I downloaded the sources, unzipped them, and put them in the site-packages folder. It worked, now I can see them from VS.
When I try to use these modules however the program crashes, this is what I have done:
from suds import *
from suds.client import Client
class Utils(object):
def doStuff(addr, mail):
cl = Client(addr, verify=False)
cl.service.authenticate(email)
cl.service.otherFunctions(...)
I know that not every module works with IronPython, so my question is: have I made some error? Or is it suds that doesn't work with IronPython? If the second, do you know of alternatives?
Thank you.

How to include third party Python packages in Sublime Text 2 plugins

I'm writing a sublime text 2 plugin that uses a module SEAPI.py which in itself imports the requests module.
Since sublime text 2 uses it's own embedded python interpreter, it doesn't see the requests module installed in my ubuntu machine (I get the following error: ImportError: No module named requests).
Best solution I could find so far was to copy the 'requests' module (the whole directory of files) from /usr/lib/python2.7/dist-packages/requests into my plugin directory in the sublime text packages dir.
But after that, it says that it can't find the 'urllib3' module.
Is there a better way to import the requests module so that I won't have to copy all the files into my plugin directory ?
The current code I'm using is as follows:
MyPlugin.py
import sublime
import sublime_plugin
import SEAPI
...
SEAPI.py
import requests
try:
import simplejson as json
except:
import json
from time import time, sleep
...
Edit:
The selected answer is correct and fixes my main question, but a different problem exists with using the current version of 'Requests' with the embedded sublime text 2 interpreter. ST2's python is missing various modules which exist in regular 2.7 python (such as 'fileio').
I've solved it with using the 'Requests' module from here:
https://github.com/bgreenlee/sublime-github
And I had to edit the 'urllib3/response.py' file to this:
try:
from cStringIO import StringIO as BytesIO
except ImportError:
pass # _fileio doesn't seem to exist in ST's python in Linux, but we don't need it
You need to bundle full requests distribution with your Python package and then modify Python's sys.path (where it looks for modules) to point to a folder containing requests folder.
Download Requests library from a PyPi and extract it manually under your plugin folder
Before importing requests in your plugin, append the corrcet folder to sys.path to point a folder where it can found requests import
The (untested) code should look like something like this:
import sys
import os
# request-dists is the folder in our plugin
sys.path.append(os.path.join(os.path.dirname(__file__), "requests-dist"))
import requests
This also assumes that requests setup.py does not do any hacks when you install the module using easy_install or pip.
You also could import requests zip directly as Python supports importing from ZIP files, assuming requests is distributed in compatible way. Example (advanced):
https://github.com/miohtama/ztanesh/blob/master/zsh-scripts/python-lib/zipimporter.py
More about sys.path trick (2004)
http://www.johnny-lin.com/cdat_tips/tips_pylang/path.html
Mikko's answer is good, but I may have found a slightly easier way:
import MyAwesomePlugin.requests
"MyAwesomePlugin" being the name of your plugin, of course.

my imports are underlined in red even though they work in Eclipse pydev

In pydev I have a python package called webcrawler. This package is in the directory '/home/raido/Workspace/WebCrawler' The package contains a number of modules; website, tier, referrer, etc. etc... Each module contains a series of functions. I wanted to use one of these functions in another pydev project so I typed....
import sys
sys.path.append('/home/raido/Workspace/WebCrawler')
from webcrawler import website
print website.getXmlLang('http://www.google.com')
The script runs fine and runs the function that prints out the information. What I don't understand is why the word website in the "from webcrawler import website" line is underlined in red. The error says...
Unresolved import: website
website Found at: TestUrl
from webcrawler import website
However, everything appears to run fine. Is this a pydev bug? How do I fix this? I tried doing it this way.
import sys
sys.path.append('/home/raido/Workspace/WebCrawler')
from webcrawler.website import getXmlLang
print getXmlLang('http://www.google.com')
Even though this also works doing it this way also underlines the import "getXmlLang" in red.
Python 2.6.5
Eclipse 3.7.1
PyDev 2.5.0.2012050419
Ubuntu 10.04
You should add all libraries used by your project in PyDev - PYTHONPATH/External Libraries tab which you can find in project's properties. This should solve the problem.

Categories