We Keep Coding

koalas pip install fails on pyarrow dependency

I tried installing Databricks' new koalas package using the recommended pip install koalas on but it failed on the pyarrow install.
I then installed pyarrow and retried koalas but it still failed on pyarrow. I visited the Github page which informed me:
If this fails to install the pyarrow dependency, you may want to try
installing with Python 3.6.x, as pip install arrow does not work out
of the box for 3.7 https://github.com/apache/arrow/issues/1125.
I searched through the discussions and could not make sense of the "solutions", perhaps because there aren't any. I am using Python 3.7.3. The error messages I get are:
creating build/temp.macosx-10.7-x86_64-3.7
-- Runnning cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/anaconda3/bin/python -DPYARROW_BOOST_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /private/tmp/pip-install-uhdr9agf/pyarrow
unable to execute 'cmake': No such file or directory
error: command 'cmake' failed with exit status 1
----------------------------------------
Failed building wheel for pyarrow
Running setup.py clean for pyarrow
Failed to build pyarrow
Installing collected packages: pyarrow, koalas
Found existing installation: pyarrow 0.13.0
Uninstalling pyarrow-0.13.0:
Successfully uninstalled pyarrow-0.13.0
Running setup.py install for pyarrow ... error
Complete output from command /anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-install-uhdr9agf/pyarrow/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/tmp/pip-record-i7k4nwil/install-record.txt --single-version-externally-managed --compile:
...
-- Runnning cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/anaconda3/bin/python -DPYARROW_BOOST_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /private/tmp/pip-install-uhdr9agf/pyarrow
unable to execute 'cmake': No such file or directory
error: command 'cmake' failed with exit status 1
----------------------------------------
Rolling back uninstall of pyarrow
...
Command "/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-install-uhdr9agf/pyarrow/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/tmp/pip-record-i7k4nwil/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-install-uhdr9agf/pyarrow/
I have tried pip install koalas, sudo pip install koalas, and sudo -H pip install koalas and all have the same error message.
Has anyone found a solution to these errors? Or is koalas not (yet) compatible with 3.7?

you probably saw this but the github post you mentioned regarding arrow says "It does work for Python<3.7. For Python 3.7, you need to have installed the Arrow C++ packages via different means."
I was able to get koalas working on a single machine spark local mode with python 3.6 and ran the github sample script successfully ... it also specifies "pyspark>=2.4.0 is recommended"
I am sure if you try 3.6 it will work for you.
import sys
print(sys.version)
import pandas as pd
import databricks.koalas as ks
import pyarrow as pa
3.6.8
pdf = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']})
print(pdf.head())
x y z
0 0 a a
1 1 b b
2 2 b b
df = ks.from_pandas(pdf)
df.columns = ['x', 'y', 'z1']
df['x2'] = df.x * df.x
df['x2']
0 0
1 1
2 4
Name: x2, dtype: int64

pip install scikit-learn error

the error is
Command "c:\users\samuel\appdata\local\programs\python\python37\python.exe -
u -c "import setuptools,
tokenize;__file__='C:\\Users\\Samuel\\AppData\\Local\\Temp\\pip-install-
bnoak8r5\\scikit-learn\\setup.py';f=getattr(tokenize, 'open', open)
(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code,
__file__, 'exec'))" install --record C:\Users\Samuel\AppData\Local\Temp\pip-
record-4yww_x3p\install-record.txt --single-version-externally-managed --
compile" failed with error code 1 in C:\Users\Samuel\AppData\Local\Temp\pip-
install-bnoak8r5\scikit-learn\
on the website its says this
Python (>= 2.7 or >= 3.3),
NumPy (>= 1.8.2),
SciPy (>= 0.13.3).
i have python 3.7
numpy 1.14.3 and scipy 1.1.0
so is my python version to new along with my scipy version?
when i download the file and try to install it i get this error
Command "python setup.py egg_info" failed with error code 1 in C:\Users\Samuel\AppData\Local\Temp\pip-req-build-6eptglns\

I suggest you download Anaconda and use conda install as your package manager for numpy, scipy, scikit-learn, etc. I can't guarantee it will fix this problem but it most certainly will be better at what you are using scikit-learn for. It would certainly change this error because it downloads a new installation of python somewhere else within anaconda's directories. Doing this has fixed this same problem for some people in the past.
I hope this helps.

error in installing the python packages in caffe installing

Im installing caffe using this instructers (link) and refer to this i use this command for install all of requirments:
sudo pip install -r $CAFFE_ROOT/python/requirements.txt
but, unfortuantly i got this error:
Command "/usr/bin/python -u -c "import setuptools,
tokenize;__file__='/tmp/pip-build-
jseAuV/leveldb/setup.py';f=getattr(tokenize, 'open', open)
(__file__);code=f.read().replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" install --
record
/tmp/pip-brJMTP-record/install-record.txt --single-version-externally-
managed --compile" failed with error code 1 in /tmp/pip-build-
jseAuV/leveldb/
Im searched for solving this problem , somone said this error will solve by this command:
pip install --upgrade setuptools
and other commands related to upgrade setuptools ... .
can , anyone help me?
I use Jetson TX2 GPU.

after search i found that causes this problem is from leveldb library.
leveldb version requirment that wroted ir requirements.txt was leveldb>=0.19,
and according to this github page this version of leveldb not supported my architecture (ARM 64 , aarch64) , but leveldb v1.2 support this architecture , for solving this problem firs i modified requirements.txt leveldb version according this:
leveldb>=0.20
then tried installation but unfortunatly it doesnt work.
refer to this page of nvidia , digits couldent be installed on jetson tx2 , but refer to this comment:
https://devtalk.nvidia.com/default/topic/1007431/jetson-tx2/issues-installing-digits-on-tx2/post/5203559/#5203559
my problem solved!
i'm happy (:
by.

Scipy Python wheel

I have got a problem with installing Scipy on my Python 2.7 , Windows in IPython.
When I enter "pip install scipy", I have one first error message:
"Failed building wheel for scipy" and then at the end
"
Command "c:\python27\python.exe -c "import setuptools,tokenize;__file__='c:\\us
ers\\admini~1\\appdata\\local\\temp\\pip-build-e3yebj\\scipy\\setup.py';exec(com
pile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __f
ile__, 'exec'))" install --record c:\users\admini~1\appdata\local\temp\pip-mwhxl
d-record\install-record.txt --single-version-externally-managed --compile" faile
d with error code 1 in c:\users\admini~1\appdata\local\temp\pip-build-e3yebj\sci
py
"
I do not know how to solve that problem, thanks if you have any ideas

You can download the wheel from this web site:
http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
You need to pick the right one. So, for
scipy‑0.19.0‑cp34‑cp34m‑win32.whl
cp34 means it will work with python 3.4
make sure you go with this one in your case:
scipy‑0.19.0‑cp27‑cp27m‑win32.whl and whatever your windows is (32 or 64).
once downloaded, go to the command line and put:
python -m pip install scipy‑0.19.0‑cp27‑cp27m‑win32.whl (making sure it is sitting in the right directory).
That should solve the problem.

I also met this problem and I then installed a Python distribution like Anaconda instead!

Installing NumPy on Windows 8.1 with Python 2.7.x

I'm very new to Python and programming world and has been going along with tutorials from newcoder.io This Here! I have been doing as per the instructions but when I try to install NumPy I get an error.
"
error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
Command "C:\Users\HP.virtualenvs\DataVizProj\Scripts\python.exe -c "import setuptools, tokenize;file='c:\users
\appdata\local\temp\pip-build-lsj5sj\numpy\setup.py';exec(compile(getattr(tokenize, 'open', open)(file).re
.replace('\r\n', '\n'), file, 'exec'))" install --record c:\users\hp\appdata\local\temp\pip-6jei4k-record\instal
cord.txt --single-version-externally-managed --compile --install-headers C:\Users\HP.virtualenvs\DataVizProj\includ
te\python2.7" failed with error code 1 in c:\users\hp\appdata\local\temp\pip-build-lsj5sj\numpy
"
But that's not enough, I tried to install VCForPython27.msi from the given link. But still, gets the same error.
Please Help!

I recommend installing the Anaconda distribution of Python. It contains more packages than you can dream of, including numpy of course: http://continuum.io/downloads
The installation is as straightforward as installing the usual Python, no matter what OS you are on.

A good solution is to download the wheel file which match your Python version here:unofficial python binaries
And then go to the folder where the .whl file is present using the command prompt and unpack the wheel file xxxxx.whl with:
>python -m pip install xxxxx.whl
This will install the library in the Lib\site-packages folder, you can check it afterwards.