We Keep Coding

Can't use .so on Mac (has bad ELF magic)

I'm using buildozer to convert a python program to a phone app on Mac connected to an Android phone with the command line:
buildozer android debug deploy run
The previous command line runs the converted app on the connected phone.But the app crashes as soon as playsound is used. As for the methods before playsound work just fine.
When I run:
adb logcat | grep python
I get the error:
ImportError: dlopen failed: "/data/data/org.test.myapp/files/app/_python_bundle/site-packages/gi/_gi.so" has bad ELF magic
When I looked it up I found that Mac cannot use .so files.
Does anybody know how I can solve this?

Okay, so I ended up fixing this when I got a similar error. (dawg.so has bad ELF magic)
Basically, the reason I got this error was because the library ("gi" in your case) was not been read properly by the android phone when deployed and hence, was "corrupted".
The bottomline reason (for me) was that it was a C/C++ library under the hood and used Cython to be converted to a Python library. Hence, this error usually means that your library needs a custom recipe.
Steps to solve it:
In the root directory (where .buildozer folder is found), I added a folder named dawg (the library name), and then, inside dawg, I git cloned the source files of dawg. To get the source files, you can just go to the PyPi page for that library and go to the Project links -> Homepage of their GitHub site. Once cloned, you can also remove .git, .gitignore, from the source files.
Once that's done, run python3 setup.py install in the dawg directory to install and hence, "cythonize" the source files
In .buildozer/android/platform/python-for android/pythonforandroid/recipes, add a new folder named "dawg" (your library name) and then, inside /dawg, make _init_.py where you will add your custom recipe.
In _init_.py, you can add your recipe and the path to the source files. Here is a template that worked for me, but you can customize it for you as per your requirements.
from pythonforandroid.recipe import IncludedFilesBehaviour, CppCompiledComponentsPythonRecipe
import os
import sys
class DAWGRecipe(IncludedFilesBehaviour, CppCompiledComponentsPythonRecipe):
version = '0.8.0'
src_filename = "../../../../../../../dawg"
name = 'dawg'
# Libraries it depends on
depends = ['setuptools']
call_hostpython_via_targetpython = False
install_in_hostpython = True
def get_recipe_env(self, arch):
env = super().get_recipe_env(arch)
env['LDFLAGS'] += ' -lc++_shared'
return env
recipe = DAWGRecipe()
Don't forget to alter the buildozer.spec. To its p4a.local_recipes, add the local path to /.buildozer/android/platform/python-for-android/pythonforandroid/recipes as that's where we add our recipes.
Clean the previous build by running buildozer android clean
Lastly, run buildozer -v android debug deploy run to build the app on android phone again.
Hope this helps :)

Downloading all jupyter notebooks from Coursera [tar size exeeding 100MB]

As mentioned in the coursera help articles in order to download notebooks from the class we need to zip all the content of root folder into single file and download the final workspace.tar.gz using these steps: but it is not working all courses.
Anyone knows proper way to do this !!

Open the home folder of your coursera jupyter notebook:
you can do this by opening any of the course notebooks and thanm selecting file> open or by clicking on Jupyter icon at the top left corner of notebook.
Open terminal inside the notebook:
On the home page of your notebooks, at the top left corner select new> terminal
Check in which dir you are:
this is important as different courses have their materials in different dir!
Some courses have a dir name jovyan and inside that you have two folders generally work and work-ro.
in work you have your actual content that you can see on your notebook home page.
in work-ro you have only read_only folder. This same folder you have it in your work dir but you cant open the content of that folder after downloading! (I dont know why I cant open it)
I turns out that this folder contains images which are in your notebooks. that is the reason you will have to zip both these folders.
Its not necessary that all the course have this folder named work!
In some courses materials are directly inside root dir. In such cases you can find the directory with your material by finding folder name ending with -ro
Ex in one of my course I located a folder named TF-ro and there was another folder named TF containing all course material! As per above pattern TF-ro contained read_only folder.
Just in case you are wondering how to navigate inside terminal: [Use these commands]
ls: list everything inside the folder
cd: to change the folder you are currently in
Ex: cd .. #go to previous folder cd <dirname> #go to that specified folder
compress both the folders using tar:
Navigate to the folder which contains both of these folders i.e work and work-ro or if you read my second case than Tf and TF-ro or folders in your case.
Use this to make tar file:
Use this when your folder contains only two dirs that you want
tar -czvf <choose a name>.tar.gz <address of dir to compress>
Ex: tar -czvf data.tar.gz ./
use this when you are in root folder and you have multiple dir along with the folders you want
tar -czvf <choose a name>.tar.gz <dir1 addres> <dir2 addres>
Ex: tar -czvf data.tar.gz ./work ./work-ro
Just in case you are wondering!
./ means current folder.
Check the size of your tar file:
This is also important!!
If your process of making tar file is taking too long or your terminal appears to be frozen ! than there are some big files in your home folder.
You can check the size of your tar file using: ls -lh data.tar.gz.
Normally the size should not be more than 10 - 15 Mbs.
If your size is in GBs than you are mostly downloading large amount of datasets and csv files!
you cannot download big files like this!
[Workaround for this problem are mentioned below]
run this command: du
This will list all the dir's and the size of dir's in current folder.
Figure out which folder has more size.
Note: size shown in this commands are in Number of sections occupied 1 section = 1024 bytes
Exclude these folder wile making tar...
In order to remove previous tar file run rm data.tar.gz
make the tar like this:
tar -czvf <yourName>.tar.gz --exclude=<address to exclude> <dir/dirs to zip>
Ex: tar -czvf data.tar.gz --exclude=./work/data --exclude=./work/- ./work ./work-ro
Move the file :
You can only see the content in the work folder (or any other folder your content is in) on your class's notebook home folder.
This is why we will move over tar file to that folder.
move using this command mv <file name> <location> Ex :mv data.tar.gz ./work
Download your file:
Now you can see your file in your home folder in your browser. simply select the file you will see download option available at the top !!
Sometimes you dont see the download button on the top, in such cases...
right click your file> save link As> then save it with .tar.gz extension
Just to confirm check the size of file you have downloaded and one in your classroom!!
Work Around for downloading big data sets:
Your course generally does not use all the csv's or data sets that it has stored in the data folder. When you do the assignments see which files are / data sets are used and download only those manually. i.e opening that file on your classroom and downloading it using using file> download
if you still want the entire thing than make separate tar file of that folder only. Than split the tar file (you will find it online easily) and than download as I have mentioned earlier!
After the download it is necessary to concatenate the files:
cat allfiles.tar.gz.part.* > allfiles.tar.gz
I would suggest not to waste time in doing this!! Just download what is required and that's it!!
I hope this was helpful !! cauz I spent 5 hr figuring out how to do it !! ENJOY !!

Alternatively, you could initialize a git repo and push it to your GitHub account.
Open terminal (Jupiter home > new > terminal)
Run the following code: (I'm assuming you've already created a GitHub repo, if not create one and then do the following; you'll need the link to your repo)
git init
git config --global user.name "test"
git config --global user.email "test"
git add -A; git commit -m "commit"
git remote add origin <_your-github-repo-url_>
git push origin master -u --verbose

You can just compress all the programming exercise (notebook + data) by placing this commands at the beginning of your notebook:
import os
!tar chvfz notebook.tar.gz *
print("File size: " + str(os.path.getsize("notebook.tar.gz")/1e6) + " MB")
if os.path.getsize("notebook.tar.gz")/1e6 >100 :
print("Splitting file")
!split -b 100M notebook.tar.gz "notebook.tar.gz."

Makefile on Windows 10 - file not found

After some time I finally managed to successfully install python and pip and run it on my machine using Visual Studio Code.
I am working in virtual environment in python and we have a Makefile with following statement:
test:
source .env && PYTHONPATH=. PY_ENV=testing py.test ${ARGS} --duration=20
File .env lives in the main directory next to Makefile. It contains some environmental variables needed for testing certain APIs.
When I take the line out of the file and run it in my terminal, everything works fine and all tests are running etc.
However if I call the following: make test I am getting this error:
$ make test
source .env && PYTHONPATH=. PY_ENV=testing py.test --duration=20
/usr/bin/sh: line 0: source: .env: file not found
make: *** [test] Error 1
(venv)
To me it looks like when running this command from within Makefile it can't see the .env file but have no idea how to solve it.

The source command isn't looking up the file in the current working directory. As mentioned in man source:
Read and execute commands from filename in the current shell
environment and return the exit status of the last command executed
from filename. If filename does not contain a slash, filenames in
PATH are used to find the directory containing filename.
Change the file path like so:
test:
source ./.env && PYTHONPATH=. PY_ENV=testing py.test ${ARGS} --duration=20
Note that this error does not occur in bash version < 4. This is due to an implementation bug when run under POSIX mode (what make uses, since its default shell is sh, which is usually bash --posix). The correct behaviour was first mentioned in the documentation of bash-2.05 (revision 28ef6c31, file doc/bashref.info):
When Bash is not in POSIX mode, the current directory is searched if
FILENAME is not found in `$PATH'.
These older versions searched the current directory regardless of POSIX mode. It was only in bash-4.0-rc1 (revision 3185942a, file general.c) that this was corrected. Running git diff 3185942a~ 3185942a general.c outputs this section:
## -69,6 +69,7 ## posix_initialize (on)
if (on != 0)
{
interactive_comments = source_uses_path = expand_aliases = 1;
+ source_searches_cwd = 0;
}

Setting environment variables accessible to program in root

I'm encountering an issue with my program not finding the environment variables when it is run as root. I currently have the program do:
#!/usr/bin/python3
from i2clibraries import i2c_adxl345
#various other imports
euid=os.geteuid()
if euid != 0:
args=['sudo',sys.executable]+sys.argv+[os.environ]
os.execlpe('sudo',*args)
#rest of program
Along with the environment variables, the necessary files are located in the directory this program is located in /home/pi/project-test
How do I set the environment variables to be accessible to this program when it restarts itself as root? They are:
export QUICK2WIRE_API_HOME=~/project-test/quick2wire-python-api
export PYTHONPATH=$PYTHONPATH:$QUICK2WIRE_API_HOME
Doing those exports within the directory of my program fixes it when run as user (pi) but not for root. Can I fix the QUICK2WIRE_API_HOME location above or do I need to load all of my libraries and programs into another location?
Note: substituting 'sudo -E' or 'sudo -E su' in does not carry the environment variables set as I expected. By the way, the actual error received is:
ImportError: No module named quick2wire.i2c
which is what the environment variables set the path to. Also, it has to be run as root as some of the program accesses the GPIO's and running as user (pi) returns:
RuntimeError: No access to /dev/mem
I've also toyed with the idea of breaking it up using multiprocessing or threading but both are just too far above my head at the moment to understand what I need to include and where to set functions, args, etc.
The complete program can be found here.
UPDATE: I redownloaded all the pertinent files into /root via Thinkbowl. Still doesn't want to accept the quick2wire library which is located in /root. Currently /root looks like:
. .config .gvfs quick2wire-python-api
.. .dbus i2clibraries .Xauthority
.bash_history .Desktop .idlerc .xsession-errors
.bashrc .gpio.sh .local
.cache .gstreamer-0.10 .profile
with i2c* and .quick* being the two libraries downloaded from thinkbowl. When I perform:
env | grep quick2wire
in /root I get:
OLDPWD=/root/quick2wire-python-api
QUICK2WIRE_API_HOME=/root/quick2wire-python-api
PYTHON=$PYTHONPATH:$QUICK2WIRE_API_HOME:/root/quick2wire-python-api

In Python (or Mac OS): two different files recognized as same file and gets re-written

Simplified description of problem, I have:
dir and file structure:
./top.py
./Task/Task.py (was deleted and then renamed as:)
./Task/task.py
top.py import task.py. But then when I run top.py, I saw that the task.py automatically changed to Task.py.
Why is this happening?
(I use mvim as the editor on Mac OS 10.6.)

The default file system of Mac OS X is case insensitive, but case preserving, as Charles said (this file system is called Mac OS Extended, or HFS Plus, and is configured by default as being case insensitive). So, in effect, ./Task/Task.py and
./Task/task.py are the same file (you can read Task.py with open('task.py'), in Python, for instance), but the name that was used upon creating the file is kept with the original case.
If you really need to distinguish between Task.py and task.py, you might want to create a case-sensitive disk image and work from it (this issue is discussed on the web).

The case of file on OSX will be preserved to whatever the file was originally created with. Hence mv Task.py task.py won't work. If you wish to rename the file you generally need to rename it to something else first. You can do this at command line by doing:
cd Task
mv Task.py temp.py
mv temp.py task.py
P.S. If your files are under git revision control it gets even more complicated.