I'm trying to use data from CSV files in PySpark. I found a module called PySpark-CSV which does exactly what I need. According to the PySpark-CSV GitHub page, "no installation [is] required", so I figured I could just unzip the source in a directory called 'pyspark_csv' in my Python path and run the commands listed on their website:
import pyspark_csv as pycsv
sc.addPyFile('pyspark_csv.py')
But this renders me with an import error saying that it can't find pyspark_csv.
The README doesn't help me any further and other info is scarce. Anyone here familiar with the module?
It means Python cannot find pyspark_csv.py. This is because you put the file in pyspark_csv and Python unaware that. Let say a full path of the directory is `/foo/pyspark_csv'. You can modify PYTHONPATH, or use other methods to inform Python where have you put your files.
#Run this in bash shell before you excute python
#Or put thisline in a bottom of .bashrc file.
export PYTHONPATH=$PYTHONPATH:/foo/pyspark_csv
Use a full path for Spark, too:
sc.addPyFile('/foo/pyspark_csv/pyspark_csv.py')
Related
I have strcture of my program like this :
I want to import some functions in the examples/borrow.py file from src/v1/lend/costants/testnetCostants.py file and from src/v1/lend/borrow.py file how can I do this in a proper way ?
I try this structure but it didn't work :
from src.v1.lend.borrow import prepareAddEscrowTransactions, prepareBorrowTransactions, prepareRepayTransaction
from src.v1.lend.costants.testnetCostants import TestnetOracle, TestnetReserveAddress, TestnetTokenPairs
from config import algodClient, sender
I receive this error "ModuleNotFoundError: No module named 'src'"
thanks all for your help
First of all as a friendly advice, for efficient code management, please use a more organized file structure like this:
file structure example
Then create an venv for every practical project and install appropriate reqiurements. Check the notebook.
After all if you use VScode, change the interpreter to your specific env:
right bottom corner of your vscode
After this, you should not see any warning in your VScode for imports if you
enter correct path and install packages completely.
Finally for running:
In your terminal, activate the project env (if your packagemanager is conda, use conda activate your_env_name)
Go to your project directory and use export PYTHONPATH=${PWD}
Then use python src/utils/main.py or your own file path.
it would be better to mention that your modular imports have no problem, if your names and structure are correct by the way. Check the link if it is helpful.
this is in Python, python 3.10.2 if we need specifics.
I’m using the Mac Terminal and I want to access a folder from it
I have a folder on my desktop that has a bunch of my modules. I wanna access these from the terminal in a style like: “import module” with using some command to get to that folder.
Help would be appreciated!
You can import sys, then sys.path.append(path_to_desired_modules). sys.path is a list of the directories where Python will search for the modules to import, so if you append the target directory to that, it should be retrievable.
How do I get the full path of the current file's directory?
Yes, I already tried these methods. They all give me the folder that contains Blender.exe
which is D:\program\Blender 2.90
And what I supposed to get is C:\Users\username\AppData\Roaming\Blender Foundation\Blender\2.90\scripts\addons\someAddonFolder right?
Is something changed in version 2.9?
Without the Blender environment, I mean run under normal Python. Methods from that topic all work well.
Did I miss something?
The old way I made the script work is by putting .blend .json .py all together in the same folder.
and refer to the relative path using //
bpy.utils.user_resource('SCRIPTS', "addons")
use this code can get you the path your custom addon you install at.
for me, it's 'C:\Users\Naoki\AppData\Roaming\Blender Foundation\Blender\2.90\scripts\addons'
found the answer over here:
https://blender.stackexchange.com/questions/20850/how-to-automatically-get-a-add-on-folders-path-regardless-of-os
I am a novice and I am trying to follow this code: https://github.com/llSourcell/autoencoder_demo/tree/master
Now, as you can see in the picture, the main file throws a no module found error. The question I have is how can I import the input_data found in the adjoint file?
You can do the following to fix the problem.
Extract the files to some folder. Open a command prompt from that folder and then run the script. Make sure that you run cmd from the directory where files are. This is because, python looks at the current directory and Python path, when you try to import a module.
Not a major issue but just an annoyance I've come upon while doing class work. I have my Notepad++ set up to run Python code straight from Notepad++ but I've noticed when trying to access files I have to use the full path to the file even given the source text file is in the same folder as the Python program being run.
However, when running my Python program through cmd I can just type in the specific file name sans the entire path.
Does anyone have a short answer as to why this might be or maybe how to reconfigure Notepad++?
Thanks in advance.
The problem is that your code is assuming that the current working directory is the same as the script directory. This is not true in general. Of course it is true if you're in a cmd window, and you cd to the script directory before running it.
If you don't want to rely on that (e.g., because you want to be able to run scripts from Notepad++, or directly from Explorer), what you want to do is use the script directory explicitly. For example:
import os
import sys
scriptdir = os.path.abspath(os.path.dirname(sys.argv[0]))
with open(os.path.join(scriptdir, 'myfile.txt')) as f:
# etc.
If you have a ton of files that your scripts reference in a ton of places, it might be better to explicitly set the working directory. Just add one line:
os.chdir(scriptdir)
For anything beyond quick&dirty scripts, it's usually better to build an installable package and use pkg_resources to access the data files. Read the Tutorial on Packaging and Distributing Projects for more details. But as long as you're only hacking up scripts to help you maintain your specific system, the scriptdir solution is workable.
In the properties of the shortcut that you use to start Notepad++, you can change its working directory, to whichever directory you're more accustomed to starting from in Python. You can also begin your python program with the appropriate os.chdir() command.