We Keep Coding

ansible: local test new module with Error:Module unable to decode valid JSON on stdin. Unable to figure out what parameters were passed

I'm new to Python. This is my first Ansible module in order to delete the SimpleDB domain from ChaosMonkey deletion.
When tested in my local venv with my Mac OS X, it keeps saying
Module unable to decode valid JSON on stdin. Unable to figure out
what parameters were passed.
Here is the code:
#!/usr/bin/python
# Delete SimpleDB Domain
from ansible.module_utils.basic import *
import boto3
def delete_sdb_domain():
fields = dict(
sdb_domain_name=dict(required=True, type='str')
)
module = AnsibleModule(argument_spec=fields)
client = boto3.client('sdb')
response = client.delete_domain(DomainName='module.params['sdb_domain_name']')
module.exit_json(changed = False, meta = response)
def main():
delete_sdb_domain()
if __name__ == '__main__':
main()
And I'm trying to pass in parameters from this file: /tmp/args.json.
and run the following command to make the local test:
$ python ./delete_sdb_domain.py /tmp/args.json
please note I'm using venv test environment on my Mac.
If you find any syntax error in my module, please also point it out.

This is not how you should test your modules.
AnsibleModule expects to have specific JSON as stdin data.
So the closest thing you can try is:
python ./delete_sdb_domain.py < /tmp/args.json
But I bet you have your json file in wrong format (no ANSIBLE_MODULE_ARGS, etc.).
To debug your modules you can use test-module script from Ansible hacking pack:
./hacking/test-module -m delete_sdb_domain.py -a "sdb_domain_name=zzz"

importerror watson_developer_cloud Python

I have pip installed watson-developer-cloud, on python v3.5
I am simply trying to run one of the example codes: alchemy_data_news_v1.py
Link:https://github.com/watson-developer-cloud/python-sdk/tree/master/examples
import json
from watson_developer_cloud import AlchemyLanguageV1
alchemy_data_news = AlchemyDataNewsV1(api_key='api-key')
results = alchemy_data_news.get_news_documents(start='now-7d', end='now',
time_slice='12h')
print(json.dumps(results, indent=2))
results = alchemy_data_news.get_news_documents(
start='1453334400',
end='1454022000',
return_fields=['enriched.url.title',
'enriched.url.url',
'enriched.url.author',
'enriched.url.publicationDate'],
query_fields={
'q.enriched.url.enrichedTitle.entities.entity':
'|text=IBM,type=company|'})
print(json.dumps(results, indent=2))
I have also tried utilizing my own personal api-key and the result is the same:
File "c:\users\Joseph Sansevero\desktop\test.py", line 2, in
watson_developer_cloud import AlchemyLanguageV1 ImportError: No module
named watson_developer_cloud

Change your import statement to
from watson_developer_cloud import AlchemyLanguageV1
Alchemy language is a different api than AlchemyNews.
Head over to https://www.ibm.com/watson/developercloud/alchemydata-news/api/v1/?python#methods and you'll see the example has AlchemyNews imported.
Also make sure you install these packages using before running your code.

Custom Module Import Issue

I can't seem to import my own custom NYT module. My project structure is as follows and I'm on a mac:
articulation/
articulation/
__init__.py # empty
lib/
nyt.py
__init__.py # empty
tests/
test_nyt.py
__init__.py # empty
When I try running python articulation/tests/test_nyt.py from that first parent directory, I get
File "articulation/tests/test_nyt.py", line 5, in <module>
from articulation.lib.nyt import NYT
ImportError: No module named articulation.lib.nyt
I also tried
(venv) Ericas-MacBook-Pro:articulation edohring$ Python -m articulation/tests/test_nyt.py
/Users/edohring/Desktop/articulation/venv/bin/Python: Import by filename is not supported.
test_nyt.py
import sys
sys.path.insert(0, '../../')
import unittest
#from mock import patch
# TODO: store example as fixture and complete test
from articulation.lib.nyt import NYT
class TestNYT(unittest.TestCase):
#patch('articulation.lib.nyt.NYT.fetch')
def test_nyt(self):
print "hi"
#assert issubclass(NYT, Article)
# self.assertTrue(sour_surprise.title == '')"""
nyt.py
from __future__ import division
import regex as re
import string
import urllib2
from collections import Counter
from bs4 import BeautifulSoup
from cookielib import CookieJar
PARSER_TYPE = 'html.parser'
class NYT:
def __init__(self, title, url):
self.url = url
self.title = title
self.words = get_words(url)
def get_words(url):
cj = CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
p = opener.open(url)
soup = BeautifulSoup(p.read(), PARSER_TYPE)
# title = soup.html.head.title.string
letters = soup.find_all('p', class_='story-body-text story-content')
if len(letters)==0:
letters = soup.find_all('p', class_='paragraph--story')
if len(letters)==0:
letters = soup.find_all('p', class_='story-body-text', )
words = Counter()
for element in letters:
a = element.get_text().split()
for c in a:
c = ''.join(ch for ch in c if c.isalpha())
c = c.lower()
if len(c) > 0:
words[c] += 1
return words
def test_nyt():
china_apple_stores = NYT('title_test', 'http://www.nytimes.com/2016/12/29/technology/iphone-china-apple-stores.html?_r=0')
assert(len(china_apple_stores.words) > 0)
# print china_apple_stores.words
fri_brief = NYT('Russia, Syria, 2017: Your Friday Briefing', 'http://www.nytimes.com/2016/12/30/briefing/us-briefing-russia-syria-2017.html')
assert(fri_brief.title == 'Russia, Syria, 2017: Your Friday Briefing')
assert(fri_brief.url == 'http://www.nytimes.com/2016/12/30/briefing/us-briefing-russia-syria-2017.html')
assert(len(fri_brief.words) > 0)
vet = NYT('title_test', 'http://lens.blogs.nytimes.com/2017/01/03/a-love-story-and-twins-for-a-combat-veteran-amputee/')
assert(len(vet.words)>0)
print "All NYT Tests Passed"
#test_nyt()
I've tried the following and none seem to work - does anyone know how to fix this?
- Adding an init.py file to the top directory -> Doesn't help
- Entering Memory Python couldn't find this - maybe because I'm using Python 2. If this is the issue I can post more what I tried.
- Adding sys.path at the top from suggestion below

Doing this:
import sys
sys.path.insert(0, '../../')
is usually a bad idea. Sometimes it's useful for when you're testing something, or you have a single-use program that you just need to work for a short time and then you're going to throw away, but in general it's a bad habit to get into because it might stop working once you move directories around or once you give the code to someone else. I would advise you not to let yourself get in the habit of doing that.
The most likely reason to get the kind of error you're seeing is that the directory /Users/edohring/Desktop/articulation does not appear in sys.path. The first thing to do is see what actually is in sys.path, and one good way to do that is to temporarily put these lines at the top of test_nyt.py:
import os.path, sys
for p in sys.path:
print(p)
if not os.path.isabs(p):
print(' (absolute: {})'.format(os.path.abspath(p)))
sys.exit()
Then run
python articulation/tests/test_nyt.py
and look at the output. You will get a line for each directory path that Python looks in to find its modules, and if any of those paths are relative, it will also print out the corresponding absolute path so that there is no confusion. I suspect you will find that /Users/edohring/Desktop/articulation does not appear anywhere in this list.
If that turns out to be the case, the most straightforward (but least future-proof) way to fix it is to run
export PYTHONPATH=".:$PYTHONPATH"
in the shell (not in Python!) before you use Python itself to do anything using your module. Directories named in the PYTHONPATH environment variable will be added to sys.path when Python starts up. This is only a temporary fix, unless you put it in a file like $HOME/.bashrc which will get read by the shell every time you open up a Terminal window. You can read about this and better ways to add the proper directory to sys.path in this question.
Perhaps a better way to run your script is to use the shell command
python -m articulation.tests.test_nyt
This needs to be run in the directory /Users/edohring/Desktop/articulation, or at least that directory needs to appear in sys.path in order for the command to work. But using the -m switch in this way causes Python to handle how it sets up sys.path a little differently, and it may work for you. You can read more about how sys.path is populated in this answer.

From *folder_name* import *variable* Python 3.4.2

File setup:
...\Project_Folder
...\Project_Folder\Project.py
...\Project_folder\Script\TestScript.py
I'm attempting to have Project.py import modules from the folder Script based on user input.
Python Version: 3.4.2
Ideally, the script would look something like
q = str(input("Input: "))
from Script import q
However, python does not recognize q as a variable when using import.
I've tried using importlib, however I cannot figure out how to import from the Script folder mentioned above.
import importlib
q = str(input("Input: "))
module = importlib.import_module(q, package=None)
I'm not certain where I would implement the file path.

Repeat of my answer originally posted at How to import a module given the full path?
as this is a Python 3.4 specific question:
This area of Python 3.4 seems to be extremely tortuous to understand, mainly because the documentation doesn't give good examples! This was my attempt using non-deprecated modules. It will import a module given the path to the .py file. I'm using it to load "plugins" at runtime.
def import_module_from_file(full_path_to_module):
"""
Import a module given the full path/filename of the .py file
Python 3.4
"""
module = None
try:
# Get module name and path from full path
module_dir, module_file = os.path.split(full_path_to_module)
module_name, module_ext = os.path.splitext(module_file)
# Get module "spec" from filename
spec = importlib.util.spec_from_file_location(module_name,full_path_to_module)
module = spec.loader.load_module()
except Exception as ec:
# Simple error printing
# Insert "sophisticated" stuff here
print(ec)
finally:
return module
# load module dynamically
path = "<enter your path here>"
module = import_module_from_file(path)
# Now use the module
# e.g. module.myFunction()

I did this by defining the entire import line as a string, formatting the string with q and then using the exec command:
imp = 'from Script import %s' %q
exec imp

Using BeautifulSoup in CGI without installing

I am trying to build a simple scraper in Python, which will run on a Webserver via CGI. Basically it will return a value determined by a parameter passed to it in a URL. I need BeautifulSoup to do the processing of HTML pages on the webserver. However, I'm using HelioHost, which doesn't give me shell access or pip etc. I can only use FTP. One the BS website, it says you can directly extract it and use it without installing.
So I got the tarball on my Win7 machine, used 7-zip to remove bz2 compression, and then tar compression, which gave me a bs4 folder and a setup.py file. I transferred the complete bs4 folder to my cgi-bin directory where the python script is located via ftp. My script code is :
#!/usr/bin/python
import cgitb
cgitb.enable()
import urllib
import urllib2
from bs4 import *
print "Content-type: text/html\n\n"
print "<html><head><title>CGI Demo</title></head>"
print "<h1>Hello World</h1>"
print "</html>"
But it is giving me an error:
/home/poiasd/public_html/cgi-bin/lel.py
6 import urllib
7 import urllib2
8 from bs4 import *
9
10 print "Content-type: text/html\n\n"
bs4 undefined
SyntaxError: invalid syntax (__init__.py, line 29)
args = ('invalid syntax', ('/home/poiasd/public_html/cgi-bin/bs4/__init__.py', 29, 6, 'from .builder import builder_registry\n'))
filename = '/home/poiasd/public_html/cgi-bin/bs4/__init__.py'
lineno = 29
msg = 'invalid syntax'
offset = 6
print_file_and_line = None
text = 'from .builder import builder_registry\n'
How can I use the bs4 module via CGI? How can I install but not-install it? Can I convert the BeautifulSoup I have on my PC to a nice little BeautifulSoup4.py which will contain all the code?

You are using a version of Python that doesn't yet support PEP 328 Relative Imports; e.g. Python 2.4 or older. BeautifulSoup 4 requires Python 2.7 or newer.
Presumably you cannot upgrade to a newer Python version. In that case you can try using BeautifulSoup 3; it'll have a few bugs and you'll be missing some features, but at least you can get past the syntax error.
However, I note that HelioHost does list Python 2.7 as supported.