We Keep Coding

file not found while running pyspark program

im new to pyspark and i want to lunch a pyspark program in standalone cluster, i followed the steps on this tutorial and i lunched my program using this command:
bin\spark-submit examples\src\main\python\LSI_MapReduce\LSI.py
here is the part of my code where the error is happening:
# load the dataset
rows = np.loadtxt('first.txt') <----- here
rows = sc.parallelize(rows)
mat = RowMatrix(rows)
# compute SVD
svd = mat.computeSVD(20, computeU=True)
the first steps of my code did run fine and i got this error:
in the line 200 FileNotFoundError: first.txt not found.
LSI_MapReducefolder has a file named first.txt at the same place as LSI.py
when i run my program on VScode it works perfectly.
how can i fix this error ?
i highely would appriciate any help .

Python, via Numpy, (not Spark) is trying to read the file from where you run your Python interpreter,
The word count example in the link reads the README.md file next to the bin folder, so if that's where you start the command, that's where your file needs to be. Otherwise, cd down into the example folder where your file exists
Also, Spark can read text files or csv files itself; so you shouldn't need numpy to do that

FileNotFound Error in python when running .py file from visual basic

I was trying to make a assistant which can perform simple task like shutting down the computer etc. For this i chose python and visual basic..... visual basic for displaying(frontend applicaton) and python for performing tasks(backend application). So i create a py file named main.py and created a folder named query and in it created a file named query.jarvis which can simply be opened as a text file. The vb(visual basic) program just write text into query.jarvis and then run the main.py file. When I run it manually by double clicking the main.py file it works fine(like in query was "shutdown" and after running main.py file by double clicking my computer shutdown) but when I try to run it from vb it shows the error file not found query\query.jarvis . I even tried to convert py file to exe by pyinstaller but it again showed the same error but only when I run it from vb.
*main.py()
def check(q):
#here was performing task according to query
f=open("query\query.jarvis")
#here the error occured
x=f.readlines()
d=x[0]
d=d.strip()
q=d.lower()
check(q)*
*vb.net
objWriter123.Close()
Dim objWriter As New System.IO.StreamWriter(moddir + "query\query.jarvis")
#here moddir is the directory of main.py file
objWriter.Write(UserQuery.Text)
objWriter.Close()
UserQuery.Text = ""
Process.Start(moddir + "main.py", AppWinStyle.MinimizedNoFocus)*

process.start sometimes has weird outcomes. i usually manage to fix it by adding explorer.exe into the mix.
Process.Start("explorer.exe", moddir & "\main.py")
also note this extra backslash, that you might have missed. in \main.py
note
in vb you concat strings with the & symbol instead of the + symbol.

ValueError: need more than 0 values to unpack (Python 2)

I am trying to replicate another researcher's findings by using the Python file that he added as a supplement to his paper. It is the first time I am diving into Python, so the error might be extremely simple to fix, yet after two days I haven't still. For context, in the Readme file there's the following instruction:
"To run the script, make sure Python2 is installed. Put all files into one folder designated as “cf_dir”.
In the script I get an error at the following lines:
if __name__ == '__main__':
cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
os.chdir(cf_dir)
cf = pd.read_csv(cf_file)
cf_phys = pd.read_csv(cf_phys_file)
ValueError: need more than 0 values to unpack
The "cf_file" and "cf_phys_file" are two major components of all files that are in the one folder named "cf_dir". The "cf_phys_file" relates only to two survey question's (Q22 and Q23), and the "cf_file" includes all other questions 1-21. Now it seems that the code is meant to retrieve those two files from the directory? Only for the "cf_phys_file" the columns 1:4 are needed. The current working directory is already set at the right location.
The path where I located "cf_dir" is as follows:
C:\Users\Marc-Marijn Ossel\Documents\RSM\Thesis\Data\Suitable for ML\Data en Artikelen\Per task Suitability for Machine Learning score readme\cf_dir
Alternative option in readme file,
In the readme file there's this option, but also here I cannot understand how to direct the path to the right location:
"Run the following command in an open terminal (substituting for file names
below): python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile
jobDataFile OESfile
This should generate the data and plots as necessary."
When I run that in "Command Prompt", I get the following error, and I am not sure how to set the working directory correctly.
- python: can't open file 'cfProcessor_AEAPnP.py': [Errno 2] No such file or directory
Thanks for the reading, and I hope there's someone who could help me!
Best regards & stay safe out there during Corona!!
Marc

cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
means, the python file expects few arguments when called.
In order to run
python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile jobDataFile OESfile
the command prompt should be in that folder.
So, open command prompt and type
cd path_to_the_folder_where_ur_python_file_is_located
Now, you would have reached the path of the python file.
Also, make sure you give full path in double quotes for the arguments.

Python for Data Analysis, Chapter 2, first example

I'm following along with the examples in a translated version of Wes McKinney's "Python for Data Analysis" and I was blocked in first example of Chapter 2
I think my problem arose because I saved a data file in a wrong path. is that right?
I stored a file, usagov_bitly_data2012-03-16-1331923249.txt, in C:\Users\HRR
and also stored folder, pydata-book-mater, that can be downloaded from http://github.com/pydata-book in C:\Users\HRR\Anaconda2\Library\bin.

Depends.
You might change the location you save your File or eddit the path you give to your code in Line 10. Since you're yousing relativ Paths i guess your script runs in C:\Users\HRR\Anaconda2\Library\bin, which means you have to go back to C:\Users\HRR or use an absolute Path ... or move the File, but hell you don't want to move a file every time you want to open it, like moving word files into msoffice file to open it, so try to change the Path.
And allways try harder ;)

In python open() will open from the current directory down unless given a full path (in linux that starts with / and windows <C>://). In your case the command is open the folder ch02 in the directory the script is running from and then open usagov_bitly_data2012-03-16-1331923249.txt in that folder.
Since you are storing the text file in C:\Users\HRR\usagov_bitly_data2012-03-16-1331923249.txt and you did not specify the directory of the script. I recommend the following command instead open(C:\\Users\\HRR\\usagov_bitly_data2012-03-16-1331923249.txt)
Note: the double \ is to escape the characters and avoid tabs and newlines showing up in the path.

Python 3 from CMD

I am attempting to run this .PY file from Command Prompt:
# Merge two .BSG files
# Starting block and world position are taken from the first file
# Example: "bsgmerge input.bsg output.bsg merged.bsg"
import io, sys
one = open(sys.argv[1]).readlines()
two = open(sys.argv[2]).readlines()
for n in [1,3,5,7,9,11,17,19,21,23]:
one[n] = one[n][:-1]+"|"+two[n].partition("|")[2]
open(sys.argv[3],"w").write("".join(one))
It is a program that takes a creation from the game Beseige and merges it with another saved creation so that opening the merged file results in both creations being present. If you want more details, you can read up on that here.
I am having trouble figuring out how to call this program from the command line. At first I thought the problem was me having Python 2 (it requires Python 3), so I uninstalled 2 and installed 3. This did not help.
What I am doing is entering the "python" command to pull up the Python environment within CMD, then entering the command to call the program based on the third comment in the file ("bsgmerge input.bsg output.bsg merged.bsg").
I tried using full file paths or simply changing to the correct directory before typing the "python" command and using only the file names, but so far I've had no luck.
When I am in the correct directory, then enter the Python environment, typing the command "bsgmerge 1.bsg 2.bsg M.bsg" (my existing files to be merged are 1.bsg and 2.bsg), this error occurs:
File "<stdin>", line 1
bsgmerge 1.bsg 2.bsg M.bsg
^
SyntaxError: invalid syntax
I took a Python course (which is why I used to have Python 2 on my machine) last fall, so I noticed that there is no "def" defining a function in the above code, which is something I've never encountered, so I'm thinking that is the root of my problems.
Thanks in advance for the help.

I was probably same problem with python launcher.
If you use Linux, first line shoud be:
#! /path/to/your/python/3
In Windows it some more complicated:
In registry by regedit change
HKEY_CLASSES_ROOT\Python.File\shell\open\command from "C:\Python27\python.exe" "%1" %* to "C:\Windows\py.exe" "%1" %*.
And first line of script shoud be:
#! python3
Now it shoud work properly.