We Keep Coding

My program throws an error when trying to run my function

I tried to make a function that checks if my Ubuntu Server has a specific package installed via apt list, when the condition isn't met it, in theory, the program should install any necessary dependencies for the other piece of software to work. Here's a function I wrote:
# Docker Configuration Tool
def DCT():
cache = apt.Cache()
if cache['docker-ce'].is_installed:
print("Docker and Docker-compose are installed on this system...")
print("If you don't have MySQL Server installed on your system use Docker to prepare and configure your Server")
__run_file = [ BASH COMMANDS ]
OS_MCE(__run_file)
else:
print("Docker and Docker-compose are not installed on this system!")
print("Preparing Environment for the Installation...\n")
__install_docker = [ BASH COMMANDS ]
OS_MCE(__install_docker)
An error when trying to run this function:
Traceback (most recent call last):
File "MIM.py", line 304, in <module>
NSWIT(True)
File "MIM.py", line 297, in NSWIT
Menu()
File "MIM.py", line 257, in Menu
DCT()
File "MIM.py", line 117, in DCT
if cache['docker-ce'].is_installed:
File "/usr/lib/python3/dist-packages/apt/cache.py", line 305, in __getitem__
raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named 'docker-ce'"

Try changing your if expression to check that the docker-ce key is in the cache map before trying to access it. If there isn't a key with that name, it makes sense to assume that the package isn't installed. So like this:
# Docker Configuration Tool
def DCT():
cache = apt.Cache()
if 'docker-ce' in cache and cache['docker-ce'].is_installed:
print("Docker and Docker-compose are installed on this system...")
print("If you don't have MySQL Server installed on your system use Docker to prepare and configure your Server")
__run_file = [ BASH COMMANDS ]
OS_MCE(__run_file)
else:
print("Docker and Docker-compose are not installed on this system!")
print("Preparing Environment for the Installation...\n")
__install_docker = [ BASH COMMANDS ]
OS_MCE(__install_docker)
Error messages are your friend. Read them carefully. In this case, the error was telling you precisely what was wrong, and with that, it is obvious how to fix it once you've been doing this for a while.
This is a good opportunity to learn about and understand short-circuited evaluation of logical expressions. The first clause of your conditional expression insures that the second clause isn't evaluated if it will throw the error you were seeing (well, Duh!, right?)...but if you don't fully understand why that is, it's a good thing to clearly understand. Maybe see: https://pythoninformer.com/python-language/intermediate-python/short-circuit-evaluation/

Getting an output from Python that gets executed by PHP

So my issue is that I have an AWS EC2 instance running Ubuntu server 18.04 and Apache2, when a button is pressed it posts some variables which are passed to a function which then uses shell_exec() to execute a Python script. The Python script then takes these variables (direction and angle) as command line arguments, it will then attempt to write then to a file called cmds.txt. When doing this however I get this error reported back to me:
Traceback (most recent call last):
File "/home/server/cmdWriter.py", line 45, in
main()
File "/home/server/cmdWriter.py", line 41, in main
servoCmds(direction, angle)
File "/home/server/cmdWriter.py", line 28, in servoCmds
cmdFile = open("cmds.txt", "a")
PermissionError: [Errno 13] Permission denied: 'cmds.txt'
After some looking I think this is because PHP executes as user "www-data" which doesn't have write privileges, so after looking at other questions I tried settung up permissions so that www-data has read and write privilages to the folder and the python file. For whatever reason this does not work! I've been pulling my hair out trying to get this to work, trying suggestions from several other questions, can anyone help me here?

You are getting the error with respect to 'opening' the file. You have to make sure that the permissions on the .txt file are at 755 or higher. If the server you are using is Linux, then you have two alternatives to modify the permissions:
If you have cPanel, you can use the cPanel interface and just change permissions by going to the file, click on Change Permissions and then set the permissions to 777.
or
From the linux command line, use chmod command - like this:
chmod 777 cmds.txt
I can't speak to the security issues associated with same without knowing more.
The other possibility is that you are using the 'a' mode, but the file does not already exist. If you are not certain the file exists when the command is executed, you may try 'a+' as that will create the file if one does not already exist.
Does that work?

Add local python module to the classpath in IntelliJ

I am trying to run/debug my python project from IntelliJ Ultimate 2018.1. I have defined a python SDK, a Django local server (as the project uses Django as a template language), the PYTHONPATH is properly defined, etc. If I execute
python manage.py runserver
from my MacOS terminal, the server starts normally.
When I am trying to run my IntelliJ configuration, it fails, with message:
/usr/bin/python2.7 -d /Users/my_user/dev/github/my_project/manage.py runserver 8000
Traceback (most recent call last):
File "/Users/my_user/dev/github/my_project/manage.py", line 19, in <module>
paths = importlib.import_module('settings.' + server + '.paths')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/Users/my_user/dev/github/my_project/settings/__init__.py", line 11, in <module>
paths = importlib.import_module('my_project.settings.' + server + '.paths')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named my_project.settings.local.paths
The settings.local folder does contains the __init__.py file, and also settings folder and the root folder of my project.
Doing a few hours of research over the internet I have realised that the main problem is to include local modules on the classpath, so they are available for server startup. For PyCharm, this is clearly explained here, and I was able to run the server from PyCharm. However, when trying to do the same thing in IntelliJ, this never works. Some people write about it here for example, but that didn't worked out for me.
I have tried using a system interpreter and also a virtual environment, created with PyCharm, with no luck. I am using python 2.7, Django 1.11 and MacOS High Sierra. Is this even possible with IntelliJ? I wonder why would people buy PyCharm if the same things can be accomplished (and so much more) with IntelliJ. Also, I have noticed that when I change a setting in IntelliJ run profile (the Django local server profile), this change is reflected back in PyCharm and viceversa. This seems to me like a bug from JetBrains, as two distinct apps somehow share the same config, probably due to the fact that the same project is loaded in both apps.

The problem was in fact trivial, as clarified with JetBrains support: I needed to set the parent folder of my_project as content root (/Users/my_user/dev/github/), instead of using /Users/my_user/dev/github/my_project/ directly. This was needed because the source code contained imports from module my_project which could not be resolved, as there was no subfolder my_project inside of my main project folder. I implemented this change in Project settings -> Modules view, Project settings -> Facets view and also in the run configuration.

Scrapy ImportError: No module named project.settings when using subprocess.Popen

I have scrapy crawler scraping thru sites. On some occasions scrapy kills itself due to RAM issues. I rewrote the spider such that it can be split and run for a site.
After the initial run, I use subprocess.Popen to submit the scrapy crawler again with new start item.
But I am getting error
ImportError: No module named shop.settingsTraceback (most recent call last):
File "/home/kumar/envs/ishop/bin/scrapy", line 4, in <module> execute()
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/cmdline.py", line 109, in execute settings = get_project_settings()
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/utils/project.py", line 60, in get_project_settings settings.setmodule(settings_module_path, priority='project')
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/settings/__init__.py", line 109, in setmodule module = import_module(module)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name)ImportError: No module named shop.settings
The subprocess cmd is
newp = Popen(comm, stderr=filename, stdout=filename, cwd=fp, shell=True)
comm -
source /home/kumar/envs/ishop/bin/activate && cd /home/kumar/projects/usg/shop/spiders/../.. && /home/kumar/envs/ishop/bin/scrapy crawl -a category=laptop -a site=newsite -a start=2 -a numpages=10 -a split=1 'allsitespider'
cwd - /home/kumar/projects/usg
I checked sys.path and it is correct ['/home/kumar/envs/ishop/bin', '/home/kumar/envs/ishop/lib64/python27.zip', '/home/kumar/envs/ishop/lib64/python2.7', '/home/kumar/envs/ishop/lib64/python2.7/plat-linux2', '/home/kumar/envs/ishop/lib64/python2.7/lib-tk', '/home/kumar/envs/ishop/lib64/python2.7/lib-old', '/home/kumar/envs/ishop/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib/python2.7', '/home/kumar/envs/ishop/lib/python2.7/site-packages']
But looks like the import statement is using "/usr/lib64/python2.7/importlib/__init__.py" instead of my virtual env.
Where am I wrong? Help please?

Looks like the settings in not being loaded properly. One solution would be to build an egg and deploy it in the env before starting the crawler.
Official docs, Eggify scrapy project

I suggest let python focus on the scrape task and use something else for process control. If it were me, I'd write a small bash script to run your program.
Test that the launcher script works by running it with env -i yourscript.sh because that will make sure it runs without any inherited environment settings.
Once the bash script works correctly, including setting up virtualenv etc, you could have python run that bash script, not python. You've sidestepped any strange environment issues at that point and got yourself a pretty solid launcher script.
Even better, given you have the bash script at that point, use a "proper" process controller (daemontools, supervisor...) spin up the process, restart on crash, etc.

How can I use Django with MySQL in MAMP stack?

I have difficulty especially in installing MySQLdb module (MySQL-python-1.2.3c1), to connect to the MySQL in MAMP stack.
I've done a number of things such as copying the mysql include directory and library (including plugin) from a fresh installation of mysql (version 5.1.47) to the one inside MAMP (version 5.1.37).
Now, the MySQLdb module build and install doesnt give me error.
The error happens when I'm calling 'import MySQLdb' from python shell (version 2.6).
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Any idea, what else do I need to do to make it works?
Thanks a bunch,
Robert
=========
Add the system response after using virtualenv as suggested by Hank Gay below...
(MyDjangoProject)MyMacPro:MyDjangoProject rhenru$ which python
/Users/rhenru/Workspace/django/MyDjangoProject/bin/python
After I run python in virtualenv, importing MySQLdb:
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
import sys and sys.path
>>> import sys
>>> print sys.path
['', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python26.zip', '/Library/Python/2.6/site-packages/PyXML-0.8.4-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/pydot-1.0.2-py2.6.egg', '/Library/Python/2.6/site-packages/pyparsing-1.5.2-py2.6.egg', '/Library/Python/2.6/site-packages/vobject-0.8.1c-py2.6.egg', '/Library/Python/2.6/site-packages/pytz-2010h-py2.6.egg', '/Library/Python/2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/distribute-0.6.12-py2.6.egg', '/Library/Python/2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-darwin', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/Extras/lib/python', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-tk', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-old', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages', '/Library/Python/2.6/site-packages', '/Library/Python/2.6/site-packages/PIL', '/Library/Python/2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/PyObjC', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/wx-2.8-mac-unicode']

How are you installing MySQL-Python? I just tested in a fresh virtualenv and pip install mysql-python seems to have done the trick.
UPDATE:
pip is sort of like a package manager for Python packages.
By default, pip installs to your current site-packages directory, which is on your $PYTHONPATH. This lets other libraries/applications (like Django) access it. pip also works well with virtualenv (it should; Ian Bicking wrote them both), which is a nifty library that lets you sandbox an application. This is nice because it means you can try out new things without polluting (or even needing write access to) the global site-packages directory.
It probably seems like yak-shaving right now, but I'd say it's worth the effort to get up to speed on pip and virtualenv (you may also want to look into virtualenvwrapper, but we'll skip that for now; it's just sugar for virtualenv). It will lead to a slightly more complicated deployment scenario than putting everything in the global site-packages, but for development it's really no harder, and there are lots of good guides to deploying using a virtualenv.
I'd recommend something like the following:
curl -0 http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
pip install virtualenv
virtualenv --distribute MyDjangoProject --no-site-packages
cd MyDjangoProject
source bin/activate (this activates the sandbox that virtualenv created)
pip install django mysql-python
At this point, you should have a totally functional Django+MySQL install (if I missed any steps, just comment and I'll try to add it in). You can start your Django project like this: django-admin.py startproject MyDjangoProject. cd into your project's directory, edit your settings.py file to point to your MySQL database, and run the dev server to test it out like so: ./manage.py runserver (you may need to chmod u+x your manage.py file). Voila! You should be able to access your site on localhost:8000. When you're done working on the project, you can just use deactivate to exit the virtualenv sandbox.
Try not to hold all this against Django: a lot of it is just best practices stuff for working with Python libraries. You could get by with a lot less, but this way it's more reproducible and you're less likely to accidentally mess up one of this project's dependencies when working on a different project.

I had this problem and it turned out to be due to an errant configuration:
export VERSIONER_PYTHON_PREFER_32_BIT=yes
I can't recall what I had this enabled for (some package that required 32-bit), probably related to Google AppEngine. But Setting it to 'no' solved by issues.
Otherwise I just installed everything using homebrew and pip.