We Keep Coding

Facebook scraper 'NoneType' object has no attribute 'find' while get_post

While i using facebook_scraper libraries to get post from facebook page with this code.
from facebook_scraper import get_posts
for post in get_posts('ThaiPBSFan', pages = 50):
print(post['text'][:100])
It work with few post, then error like this.
Traceback (most recent call last):
File ".\main.py", line 2, in <module>
for post in get_posts('ThaiPBSFan', pages = 50):
File "C:\Users\admin\AppData\Local\Programs\Python\Python37-32\lib\site-packages\facebook_scraper.py", line 75, in _get_posts
yield _extract_post(article)
File "C:\Users\admin\AppData\Local\Programs\Python\Python37-32\lib\site-packages\facebook_scraper.py", line 102, in _extract_post
text, post_text, shared_text = _extract_text(article)
File "C:\Users\admin\AppData\Local\Programs\Python\Python37-32\lib\site-packages\facebook_scraper.py", line 137, in _extract_text
nodes = article.find('p, header')
AttributeError: 'NoneType' object has no attribute 'find'
So what's a problem and how can i fix it.

From the traceback, it seems that facebook_scraper is not returning a valid post; this may be because there are no further posts to find on the page.
Therefore, you could use a try/except block to catch this exception, i.e.:
try:
for post in get_posts('ThaiPBSFan', pages=50):
print(post['text'][:100])
except AttributeError:
print("No more posts to get")
It's not ideal as you would preferably be able to get a more specific exception once there were no more posts to retrieve, but it should work in your case. Be careful with the code insider your try clause - if an AttributeError is raise anywhere else, you will miss it.

I had the same issue, but only when using the most recent version of the package (0.1.12). Try with an older version of the package. For example, I tried the version 0.1.4 and it worked well. To install it, write:
pip install facebook_scraper==0.1.4
in your terminal.

AttributeError: 'ElasticLoadBalancingv2' object has no attribute 'set_desired_capacity'

import boto3
import json
import time
client = boto3.client('elbv2')
desired_capacity=8
client.set_desired_capacity(
AutoScalingGroupName='Test-Web',
DesiredCapacity=desired_capacity,
HonorCooldown=True)
and
boto3==1.7.1
When I run this script I get a
File "deploy_staging_web.py", line 6, in <module>
client.set_desired_capacity(
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 601, in __getattr__
self.__class__.__name__, item)
AttributeError: 'ElasticLoadBalancingv2' object has no attribute 'set_desired_capacity'
I intended to use python to scale aws instances up and down.
I'm not inside any virtual environment at the moment.
why is it being thrown, and how do I get across it?
It is even mentioned here on the official documentation : https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.html#AutoScaling.Client.set_desired_capacity

The official document is for the latest version, not your too old version. Upgrade your boto3 package to the latest. The most recent version is 1.9.243.

The problem turns out to be a silly one.
boto3 has moved around the various functions.
set_desired_capacity is no longer part of 'elbv2' .
It is part of 'autoscaling' https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.html#AutoScaling.Client.set_desired_capacity
While 'describe_target_health' is still part of the former https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elbv2.html?highlight=elb#ElasticLoadBalancingv2.Client.describe_target_health.
Updating
client = boto3.client('elbv2')
to
client = boto3.client('autoscaling')
has solved my problem.

Error with Python import

I have problem:
I am using this package: https://github.com/ulule/django-badgify
It works perfectly, but now I need to create Custom model Badge. I have made everything, as in docs:
main.models.py:
from badgify.models.base.badge import Badge as BaseBadge
class GuidaBadge(BaseBadge):
class Meta(BaseBadge.Meta):
abstract = False
settings.py:
BADGIFY_BADGE_MODEL = "main.models.GuidaBadge"
But it cause error:
File "D:\virtenvs\codeguida\codeguida\main\models.py", line 11, in <module>
from badgify.models.base.badge import Badge as BaseBadge
File "D:\virtenvs\codeguida\lib\site-packages\badgify\models\__init__.py", line 8, in <module>
Badge = load_class(settings.BADGE_MODEL)
File "D:\virtenvs\codeguida\lib\site-packages\badgify\utils.py", line 88, in load_class
raise exceptions.ImproperlyConfigured(txt)
django.core.exceptions.ImproperlyConfigured: Backend module "main.models" does not define a "GuidaBadge" class.
That is, https://github.com/ulule/django-badgify/blob/master/badgify/utils.py#L79
It seems that Python can`t find "GuidaBadge" class. So I have tried to use function load_class() in shell - it return right class...
I think that the error cause, in this way:
As we can see in Traceback, firstly:
File "D:\virtenvs\codeguida\codeguida\main\models.py", line 11, in <module>
from badgify.models.base.badge import Badge as BaseBadge
Program asks python to import Badge class from badgify package
Then Python tried to import it, and encounter with
File "D:\virtenvs\codeguida\lib\site-packages\badgify\models\__init__.py", line 8, in <module>
Badge = load_class(settings.BADGE_MODEL)
Here program asks python to load_class from string (that is stored in settings, e.g. 'main.models.GuidaBadge')
But Python has not run this part of models, yet. And it cause error that there is not class "GuidaBadge" in "main.models".
Am I right?
How to fix it?

dont do
from badgify.models.base.badge import Badge as BaseBadge
instead do
import badgify.models.base.badge
class GuidaBadge(badgify.models.base.badge.Badge):
class Meta(BaseBadge.Meta):
abstract = False
"from" and "as" imports are having problems with circularity because of namespace changes (the from imported module is not recognized as the original model because it is imported as a different namespace).

main.models.py:
Is your file named that way or is it models.py that lives in main directory? If first one, change it to second one.
Check also that you have __init__.py file in main directory and that your main directory lives in python path.

Why would inspect.getfile give me a file that's not there?

Or, Saltstack + docker-py AttributeError: 'RecentlyUsedContainer' object has no attribute 'lock'
I have been digging into this issue to no avail. I'm trying to use SaltStack to manage my docker images/containers but ran into this problem.
Initially I was using the salt state docker.running but that presented as the command does not exist. When I changed the state to docker.running, I got the traceback I posted over at that GitHub issue:
ID: scheduler
Function: docker.pulled
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1563, in call
**cdata['kwargs'])
File "/usr/lib/python2.7/dist-packages/salt/states/dockerio.py", line 271, in pulled
returned = pull(name, tag=tag, insecure_registry=insecure_registry)
File "/usr/lib/python2.7/dist-packages/salt/modules/dockerio.py", line 1599, in pull
client = _get_client()
File "/usr/lib/python2.7/dist-packages/salt/modules/dockerio.py", line 277, in _get_client
client._version = client.version()['ApiVersion']
File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 837, in version
return self._result(self._get(url), json=True)
File "/usr/local/lib/python2.7/dist-packages/docker/clientbase.py", line 86, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 310, in get
#: Stream response content default.
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 279, in request
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 374, in send
url=request.url,
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 155, in send
**proxy_kwargs)
File "/usr/local/lib/python2.7/dist-packages/docker/unixconn/unixconn.py", line 74, in get_connection
with self.pools.lock:
AttributeError: 'RecentlyUsedContainer' object has no attribute 'lock'
Started: 09:33:42.873628
Duration: 22.115 ms
After searching Google a bit more and coming up with nothing, I went ahead and started reading the source.
After reading unixconn.py and realizing that RecentlyUsedContainer was coming from urllib3, I went and tracked down the source for that and discovered that there was a _lock attribute that was changed to lock a while ago. That seemed strange.
I looked closer at the imports and realized that unixconn.py was attempting to use requests' built-in urllib3 and then falling back to the stand alone urllib3. So I checked out the requests urllib3 and found that it did, indeed have the _lock -> lock change. But it was newer than my version of requests. So I upgraded requests and tried again. Still no dice - same AttributeError.
Now things start to get weird.
In order to get information back to my salt master, I started mucking with the docker-py and urllib3 code on my salt minion. At first I raised exceptions with urllib3.__file__ to make sure I was using the right file. But occasionally the file name that it would return was in a file and a folder that did not exist. Usually it was displaying /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/_collections.pyc, but when I would delete that file thinking that maybe the .pyc being cached was causing a problem it would still say that was the __file__, even though it didn't exist.
Then I discovered inspect.getfile. And I got the same bizarre behavior - I could delete the .pyc file and yet inspect.getfile(self.pools) would return the non-existent file.
To make life even better, I've added
raise Exception('Pining for the Fjords')
to
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/_collections.py
At the end of the RecentlyUsedContainer.__init__. Yet that exception does not raise.
And I have just confirmed that something is in fact lying to me, because despite changing unixconn.py
def get_connection(self, url, proxies=None):
import inspect
r = RecentlyUsedContainer(10)
raise Exception(inspect.getfile(r.__class__) + '\n' + r.__doc__)
which returns /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/_collections.pyc, when I go edit that .pyc and modify the RecentlyUsedContainer's docstring I get the original docstring.
And finally, when I edit /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc and change it's docstring, (or the same path but _collections.py instead)...
I still get the same docstring!
Why is the wrong code getting executed here, and how can I find out where it is so I can fix the problem?

So I finally figured out the problem:
It did have something to do with salt. For some reason the way the salt minion imported the docker-py library it did some sort of... partial hold on the imports. I suspect that what was happening was that salt was re-importing just the docker-py library specifically so when I would make changes to those files the changes would show up.
However, since the Python import mechanism will search for pre-imported modules first the urllib3 code was never re-imported.
Ultimately all that is required is to restart the salt minion:
salt 'my-minion' cmd.run "nohup /bin/sh -c 'sleep 10 && salt-call --local service.restart salt-minion'"

Google App Engine with local Django 1.1 gets Intermittent Failures

I'm using the Windows Launcher development environment for Google App Engine.
I have downloaded Django 1.1.2 source, and un-tarrred the "django" subdirectory to live within my application directory (a peer of app.yaml)
At the top of each .py source file, I do this:
import settings
import os
os.environ["DJANGO_SETTINGS_MODULE"] = 'settings'
In my file settings.py (which lives at the root of the app directory, as well), I do this:
DEBUG = True
TEMPLATE_DIRS = ('html')
INSTALLED_APPS = ('filters')
import os
os.environ["DJANGO_SETTINGS_MODULE"] = 'settings'
from google.appengine.dist import use_library
use_library('django', '1.1')
from django.template import loader
Yes, this looks a bit like overkill, doesn't it?
I only use django.template. I don't explicitly use any other part of django.
However, intermittently I get one of two errors:
1) Django complains that DJANGO_SETTINGS_MODULE is not defined.
2) Django complains that common.html (a template I'm extending in other templates) doesn't exist.
95% of the time, these errors are not encountered, and they randomly just start happening. Once in that state, the local server seems "wedged" and re-booting it generally fixes it.
What's causing this to happen, and what can I do about it? How can I even debug it?
Here is the traceback from the error:
Traceback (most recent call last):
File "C:\code\kwbudget\edit_budget.py", line 34, in get
self.response.out.write(t.render(template.Context(values)))
File "C:\code\kwbudget\django\template\__init__.py", line 165, in render
return self.nodelist.render(context)
File "C:\code\kwbudget\django\template\__init__.py", line 784, in render
bits.append(self.render_node(node, context))
File "C:\code\kwbudget\django\template\__init__.py", line 797, in render_node
return node.render(context)
File "C:\code\kwbudget\django\template\loader_tags.py", line 71, in render
compiled_parent = self.get_parent(context)
File "C:\code\kwbudget\django\template\loader_tags.py", line 66, in get_parent
raise TemplateSyntaxError, "Template %r cannot be extended, because it doesn't exist" % parent
TemplateSyntaxError: Template u'common.html' cannot be extended, because it doesn't exist
And edit_budget.py starts with exactly the lines that I included up top.
All templates live in a directory named "html" in my root directory, and "html/common.html" exists. I know the template engine finds them, because I start out with "html/edit_budget.html" which extends common.html.
It looks as if the settings module somehow isn't applied (because that's what adds html to the search path for templates).

Firstly, although django is now a LOT more compatible with app engine than it once, some major incompatibilities still exist between the two platforms, meaning that you can't just dump a stock copy of django into your appengine directory and have it work out of the box. Things will error in strange ways.
There are a number of projects which aim to improve compatibility between the two projects, the most prominent is app-engine-patch. I highly suggest reading the following article http://code.google.com/appengine/articles/app-engine-patch.html and the rest of the articles located at code.google.com/appengine/articles/ under the django tab.
as for some of you're specific problems, you could try this within your setup script:
#setup django environment
from django.core.management import setup_environ
import settings
setup_envion(settings)
this is what django uses internally for setting up the environment within manage.py and is the accepted best practice for setting up django for use with scripts (like app engine).

I'm having exactly the same issue, and I haven't been able to work around it... though I've noticed it happens a LOT less with the real GAE than it does with the development server I run on my Linux workstation.