We Keep Coding

Is there a version compatibility issue between Spark/Hadoop/Scala/Java/Python?

I'm getting an error while running spark-shell command through cmd but unfortunately without any luck so far. I have Python/Java/Spark/Hadoop(winutils.exe)/Scala installed with versions as below:
Python: 3.7.3
Java: 1.8.0_311
Spark: 3.2.0
Hadoop(winutils.exe):2.5x
scala sbt: sbt-1.5.5.msi
I followed below steps and ran spark-shell (C:\Program Files\spark-3.2.0-bin-hadoop3.2\bin>) through cmd:
Create JAVA_HOME variable: C:\Program Files\Java\jdk1.8.0_311\bin
Add the following part to your path: %JAVA_HOME%\bin
Create SPARK_HOME variable: C:\spark-3.2.0-bin-hadoop3.2\bin
Add the following part to your path: %SPARK_HOME%\bin
The most important part Hadoop path should include bin file before winutils.exe as the following: C:\Hadoop\bin Sure you will locate winutils.exe inside this path.
Create HADOOP_HOME Variable: C:\Hadoop
Add the following part to your path: %HADOOP_HOME%\bin
Am I missing out on anything? I've posted my question with error details in another thread (spark-shell command throwing this error: SparkContext: Error initializing SparkContext)

You went the difficult way in installing everything by hand. You may need Scala too, be extremely vigilant with the version you are installing, from your example it seems like it’s Scala 2.12.
But you are right: Spark is extremely demanding in term of version matching. Java 8 is good. Java 11 is ok too, but not any more recent version.
Alternatively, you can:
Try a very simple app like in https://github.com/jgperrin/net.jgp.books.spark.ch01
Use Docker with a pre made image, and if your goal is to do Python, I would recommend an image with Jupiter and Spark preconfigured together.

vs Intellicode not working for python [ vs code extension]

I am trying to get vs intellicode up and running for python and there were no errors in installation.
However, I am not seeing any starred suggestions.
I did some research and found that ReSharper may cause the problem but I dont have resharper installed.
but when i checked the output of the python language server output, it read the following:
Current invocation parsing returned null, aborting IntelliCode recommendation!
I guess thats the problem.
Any solutions to get it up and running?
Thanks, in advance.

Had same problem with python in WSL. Try to disable the deeplearning model from the extension's settings.
See https://github.com/MicrosoftDocs/intellicode/issues/191#issuecomment-647699535

Error in calling a Python script from a Basic function

I am new in Python and Basic. I am trying to replicate the IMPORTHTML function from Google Sheets in LibreOffice (LO) Calc. In a nutshell, I want to create a GetHtmTable( Url, Table Index) Basic function in Calc which will call a Python script to do the heavy work.
So based on Villeroy's great example, I implemented in LO 5.1.6.2. the Basic SOUNDEX function which calls the Python script sheetFunctions.py to get familiar with the process. My environment is Linux Mint 18, I use Python 3, I imported all kind of libraries such as Uno, PIP etc.. I use PycharmProjects as a Python editor.
I see clearly under the LO Calc menu tools->macro->organize macros->python the sheetFunctions.py Python script, which indeed is in the folder /usr/lib/libreoffice/share/Scripts/python.
Whenever, I run the SOUNDEX Basic function I see the following error message:
BASIC runtime error. An exception occurred Type:
com.sun.star.script.provider.ScriptFrameworkErrorException Message:
: an
error occurred during file opening
/usr/lib/libreoffice/program/pythonscript.py:429 in function
getModuleByUrl() [lastRead = self.sfa.getDateTimeModified( url )]
/usr/lib/libreoffice/program/pythonscript.py:993 in function
getScript() [mod = self.provCtx.getModuleByUrl( fileUri )]
I tried to debug the SOUNDEX basic function and found out the blocking point is when the program runs getScript("vnd.sun.star.script:sheetFunctions.py$soundex?language=Python&location=user").
I've been trying for days now to overcome this error, unsuccessfully I must confess.
I wonder if I need to bring some extra extensions in the Basic environment or a missing add-in in the Linux/Python one?
I changed the location=user by location=document and got stuck again. I added recently libreoffice-script-provider-python thanks to the command sudo apt-get install libreoffice-script-provider-python but this did not help. I also embedded in the Calc document the Python script but same this did not solve the issue.

The location name does not match. The standard place for self-written scripts is under the user directory. This is location=user, for example ~/.config/libreoffice/4/user/Scripts/python.
Then there is location=share, which refers to the path in your question. These parameters are described under Python Script in the URI Specification.
See also my answer to this question. Be sure to try the APSO extension if you haven't yet. Especially, APSO helps when using location=document, because embedding requires several steps including editing manifest.xml.

Got errors, while running exe file built with pyinstaller and Google Cloud API integration in python

I am working one file python project.
I integrated google-cloud-API for realtime speech streaming and recognition.
It works with python aaa.py command well.
Now I need windows build file(.exe), so I used pyinstaller program and I got aaa.exe file successfully.
But I got this error while running speech streaming by using Google cloud API.
[Errno 2] No such file or directory:
'D:\AI\ai\dist\AAA\google\cloud\gapic\speech\v1\speech_client_config.json'
So I copied this speech_client_config.json file in needed path, after that I got below error again.
Exception in 'grpc._cython.cygrpc.ssl_roots_override_callback'
ignored E0511 01:13:14.320000000 3108
src/core/lib/security/security_connector/security _connector.cc:1170]
assertion failed: pem_root_certs != nullptr
Then, I can not find solution to get working version with google-cloud API.
I am using python version 2.7.14
I need your friendly help.
Thanks.

I had the same problem. If you are willing to distribute roots.pem with your executable (just search for the file - it should be buried deep within the installation directory of grpcio), I had luck fixing this by setting GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable to the full path of this roots.pem file.

Update 2021
To anyone who is experiencing this issue. I got it working thanks to these amazing people. See the full conversation on this github issue.
Here is the link
Step 1
Credits to #cbenhagen & #rising-stark on this github link.
A PyInstaller hook called hook-grpc.py looking like this would do the trick:
Create a python file named hook-grpc.py with this code.
from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files('grpc')
Step 2
Put the hook-grpc.py file in your \site-packages\PyInstaller\hooks directory of the python environment you are running on. So basically you can find it at
C:\Users\yourusername\AppData\Local\Programs\Python\Python37\Lib\site-packages\PyInstaller\hooks
Note:
Just change the yourusername and Python37 to your
respective username and python version you are using.
For Anaconda users it might be different. Check this site
to find the anaconda python environment path you are using.
Step 3
Once you've done that you can now convert your .py python program to .exe using pyinstaller and it should work.

This looks to me like a SSL credentials mistake. I think you are not being allowed to GC. Check this code snippet and this documentation.

Error: 'python35_d.dll is missing' while running C++ code with Python in MS Visual Studio 2015

I am trying to embed Python in my C++ code to use WordNet through nltk. I am new to Python and I am using MS Visual Studio 2015 to do the same but I'm facing problems to use Python on MSVS 15.
So far I've,
installed Python 3.5.1,
included this path: ...\Python\Python35-32\include, which let me include the Python.h library,
added this path: ...\Python\Python35-32\libs to access the lib files, and python35.lib as dependency in Linker\Input setting of MSVS15,
then there an error occurred, demanding python35_d.lib, so I compiled Python's source in Debug mode & copied that file to that ...\libs folder and it solved the problem.
This all happened when I was trying to run this only-
Py_Initialize();
PyRun_SimpleString("print (\"Hello World!\\n\")");
Py_Finalize();
Now I'm trying to run a Python function from a .py file. I'm trying to run this from the Python tutorials but I'm getting this error -
While debugging, I have found that when this macro gets called: Py_DECREF(), this error occurs.
Do I only need to copy python35_d.dll from PCBuild\win32 to C:\Windows\[System|System32|SysWOW64]\ to solve the problem as I don't know if it'll have any side-effects?
Although the file is already present in ...\Python\Python35-32\DLLs\.
Is there a proper setup to be done before running Python in C/C++ using MSVS, like this for Unix-like systems? (I mean by adding other dependencies or folders to the settings)
I've been reading Python documentation continuously and even other stackoverflow questions but I'm still struggling a lot (I don't know why) even to do simple things.
Can somebody guide/enlighten me for something I may be missing because I'm continuously getting errors here & there, please?
Thanks.

copy python35.dll into the folder where .sln file is present