We Keep Coding

Failing import of zipped library with py-files

I have to maintain an oll code running with pyspark.
It's using a method I've never seen.
I have some reusable code zipped into a file ingestion.zip.
Then, this file is called using a pipeline.cfg file like this:
[spark]
master=spark://master
py-files=${HOME}/lib/ingestion.zip
spark-submit=${SPARK_HOME}/bin/spark-submit
When I'm trying to import the library as shown below, I cant make Pycharm understand that the lib should point to the zip file.
from ingestion.data import csv, storage
I've seen the zip is a solution proposed by spark-submit using py-files but how can I make it running on my IDE ?

I haven't used below method with pycharm, but it worked for us with spark-submit and we could import these modules using normal import statements.
Actually, we had a very few files to import and we needed something quick. So, if you also have the same use-case and if pycharm allows then maybe, you can give it a try.
--py-files s3://bucket-name/module1.py,s3://bucket-name/module2.py,s3://bucket-name/module3.py,s3://bucket-name/module4.py"
(Note - there shouldn't be any spaces.)
(Note - this suggestion is only an interim solution till someone replies with a better answer.)

Can't pass data from octave to python through command line arguments

I am calculating HoG feature descriptors in Octave and then I am trying to cluster those data in Python using scikit-learn.
For testing my code in Python I am trying to pass a 4000x2 data to Python.
I am calling the Python script from Octave using
system('python filename.py data')
and then trying to get the data using
sys.argv
but I am getting the second argument as a string 'data' and not the 4000x2 data that I am passing from Octave
What should I do so that I can get the original data in Python and not just the string 'data'

There's a python command built into octave.
Alternatively, I would save as a .mat file, and open this in your python script using scipy.io.loadmat.
There's also eval_py and python_cmd from the symbolic package, but I'm not sure if this is appropriate for your particular use-case. The most general, matlab-compatible, and recommended way to do this would be the .mat one.

I am not an expert regarding Octave but the answer is most likely something like that:
command=sprintf("python filename.py %s",data)
system(command)
Be careful that the amount of command line arguments is limited in most operating systems.

Python how to learn about importing files etc

I'm fairly new to python programming, i'm familiar with the very basic stuff and i'm currently learning about creating function definitions in scripts.
In particular i'm a mac user and i'm using text wrangler to write and run my python programmes.
Now that i have learned how to define basic functions in scripts i have questions which my notes do not seem to answer.
How do i import my definition which is saved in a file on my desktop to use on IDLE? I've tried
import fileaname
in IDLE and it does not work.
Secondly Suppose i create a function A in one script and then another function B in a separate script that depends on A, do i have to import A in the script for B first? Do they need to be saved in the same file?
I appreciate any advice and useful tips.

With import filename , you want to make sure its in the same directory as your original file. Try using this too
import sys
sys.path.append(directory_path)
It seems to be a common issue.

Why doesn't matlab find this function call?

So I downloaded some compiled matlab files. I see the following files in a folder.
makemesh.mexmaci64
makemesh.mexw32
makemesh.mexw64
I added this folder to userpath, and now the path variable shows that this folder exists in it.
I try to run a test script that came with these files, and I get this error message.
Undefined function 'makemesh' for input arguments of type 'struct'.
Now to trouble shoot, I need to find,
Does it find the function, but the data type is wrong.
It doesn't even find the function.
And if it is case 1.
1a. How do I find out what is the proper data structure that function expects.
I am good with python, but new to matlab, so any tips you can add regarding how to query the help string of a function, how to print out the function signature given a function name (like ?function_name in ipython interface) would be super useful.
thank you,

computer('arch') returns glnxa64 which stands for GnuLiNuX 64bit. You have downloaded binaries for Win32bit (mexw32), Win64bit (mexw64) and IOS on Intel (mexmaci64). Either get the source files to compile it yourself or the binaries for linux.

You can find the location of a file on your path by using the which command. For example:
which makemesh.mexw64
should print the location of that file to the terminal. If you get
'makemesh.mexmaci64' not found.
then it means that the file does not exist on your path.
As for finding out what the function wants, I'd start with "help"
help makemesh.mexw64
and see if that gives you anything useful.
I suspect that somehow or other, your matlab is not recognizing the mex file as a function, so I'd start looking there.

You can use the command which <functionName> to see if Matlab is seeing your function.
Have you tried help <functionName> to see if there are any useful comments on what your function is expecting?

change matlab path from python

So I have the following problem: I have a folder at a location known from a config.py file, in my case externals/bct. Now this needs to be added to matlab's path. Now I've searched for some examples to edit matlab path but from what I can see let's say here: matlab path the changes are done from matlab. My question would then be: is there any way I could change the matlab path from python?
Best regards,
Bogdan

Thanks for the inputs. The project already has an adapter that allows runnig matlab code from python using from scipy.io import loadmat, savemat. The problem was that we are using BCT and that need to be added to matlab path dinamically at startup. The solution that worked for me was to use the method already defined to execute matlab code and just send at startup:
addpath(PATH_TO_BCT); savepath;

Your source says:
path displays the MATLAB search path, which is stored in pathdef.m
I believe your best bet is to find this pathdef.m file in the Matlab install folder, then open and change it from python.
On my Windows Machine with Matlab 2008a, it's in C:\MATLAB\R2008a\toolbox\local\pathdef.m. There are two emptied-out versions of the same file at \local\ja and \local\template, but that first one seems to be the one that counts.
There's a big warning in it saying not to edit, but it's plain text Matlab language, really easy to reverse-engineer, you should be fine. Just don't forget the semicolon at the end of each path string (unless they changed synthax in the new version you might have... just take a look at your file.)