Assume I have a script 'test.py'. To run this in Jupyter terminal I can simply put
python test.py
However, what if I wish to run this with K8s cluster rather than Python. Is there any way to execute with a specific Kernel from terminal? For example,
python --pyspark_k8s test.py
The above line is for illustrational purposes :-)
Thanks
Related
I would just like to say off the bat, that I am very new to programming and this could be a real quick fix.
I am having trouble running a python script out of a standard VS Code from my 'base' conda env: python 3.7.6, but for some reason the same code when transferred into a jupyter notebook or even ran from the makeshift jup notebook on VS CODE it works fine.
When I run the code in the standard VS Code with the same python interpreter, I continue to have the following errors:
I would really like to learn how to avoid this in the future, because I prefer to write in VS Code modules instead of out of Jup. Notebooks.
Thanks for the help!
data= pickle.load(open('myData.pickle','rb'))
_pickle.UnpicklingError: invalid load key, '\xef'.
Here I also attached the image of the jup notebook output to show that the code runs fine on its own when not in a standard script .py file. Since I am new, the best I can do is leave the link to the image.
I am using python and Visual Studio code to develop my code. I can run my scripts successfully (Using Run Python File in Terminal). However, this is very inconvenient since after each run I go back to the terminal window and do not have access to python, therefore, I will lose the variables in the previous run so I have three options: 1- Print all the variables I want to screen (This is clearly not scalable). 2- Run the file again. 3- Run the Python file in an interactive window(Ipython).
Option 3 is exactly like running the file in a Jupyter Notebook which is very convenient since you can create a cell below the current cell and keep using the variables after one run.
I would like to know what is the best method to run the python script from the terminal (Pycharm, VS code,...) and can work with variables after (not like every time you forgot to print something you run the file again?
If you set a breakpoint then debug your code (f5), on the bottom of VSCode there is a Debug Console, you can type in debug command there (print(), type(), etc). That seems like it might be what you are looking for.
Try shift + enter on each line this will open the terminal with python
Or try to run .ipynb files instead of .py files in vscode
I'm using mip4py and I am able to run my code from the command prompt using mpiexec -n 4 python myfile.py. I want to use the 'run' button in Spyder to do the same, how do I go about integrating my custom run command (or mpi) with Spyder run?
(Spyder maintainer here) You can run your file with our Run button by putting the following code in a different file:
from IPython import get_ipython
ip = get_ipython()
ip.run_cell("!mpiexec -n 4 python myfile.py")
Note: It's very important that you use a different file to not cause an infinite recursion while evaluating the above code.
I am using dryscrape in a python script. The python script is called in a bash script, which is run by cron. For those who may not be aware, dryscrape is a headless browser (use QtWebkit in the background - so requires an xsession).
Here are the main points concerning the issue I'm having
When I run the python script from the command line, it works
When I run the bash script from the command line, it works too
I figured out that this may have something to do with different environments between my command prompt and when the cron job is running, so I modified my bash script to source my .profile as follows:
#/bin/bash
. /full/path/to/my/home/directory/.profile
python script_to_run.py
This is what my cronjob crontab entry looks like:
0,55 14-22 * * 1-5 /path/to/script.sh >> $(date "+/path/to/logs/\%Y\%m\%d.mydownload.log" )
By the way, I know that the job is being run (I can see entries in /var/log/syslog, and the script also writes to a log file - which is where I get the error message below):
In all cases, I got the following error message:
Could not connect to X server. Try calling dryscrape.start_xvfb()
before creating a session
I have installed the prerequisites, on my machine (obviously - since it runs at the command line). At the moment, I have run out of ideas.
What is causing the script to run fine at the console, and then fail when run by cron?
[[Relevant Details]]
OS: Linux 16.0.4 LTS
bash: version 4.3.46(1)
cron user: myself (i.e. same user at the command prompt)
dryscrape: version 1.0.1
The solution to this was to call the dryscrape.start_xvfb() method before starting the dryscrape session.
Cron user does not have display, so you cannot run any command which requires a display.
You need to modify the python script to do not use any type of display (check carefully, because some python commands, even though they do not open any display , they internally check for this variable).
The best way to test is to ssh into the machine without Display, and check if you can run it from there without erros.
Been using the safe and easy confines of PyCharm for a bit now, but I'm trying to get more familiar with using a text editor and the terminal together, so I've forced myself to start using iPython Notebook and Emacs. Aaaaand I have some really dumb questions.
after firing up ipython notebook from terminal with the command 'ipython notebook', it pops up on my browser and lets me code. but can i not use terminal while it's connected to ipython notebook server?
after writing my code in ipython notebook, i'm left with a something.ipynb file. How do I run this file from terminal? If it was a .py file, i know i could execute it by tying python something.py from the command line; but it doesn't work if i type python something.ipynb in the command line. And of course, I assume I hit Control-C to quit out of the running server in terminal first? or do I run the command without quitting that? Or am i doomed to test it in iPython and then copy and paste it to a different txt editor like Emacs and save it in a .py file to run it?
what good is the .ipynb file if i can't run it in terminal or outside the iPython Notebook browser? Could I get my code in a .py from iPython Notebook if I wanted to? (i assume I'll be able to easily run it in terminal by tying something.py then)
thanks in advance. I'm still very much trying to figure out how to use this thing and there aren't many answers out there for questions this elementary.
Yes, you can not use the same terminal. Solutions: open another terminal or run ipython notebook inside screen. If you use Windows you might want to take a look into this question
Notebook documents (ipynb files) can be converted to a range of static formats including LaTeX, HTML, PDF and Python. Read more about converting notebooks in manual
Notebooks are great, because you can show other people your interactive sessions accompanied with text, which may have rich formatting. And if someone can run notebook server he can easily reproduce your computations and maybe modify them. Check out awesome notebook on traveling salesperson problem by Peter Norvig as an example of what you can do with ipynb. Or this notebook. More examples are available here
You can run your IPython notebook process in background.
On Unix platforms you can perform that with (note the leading &):
ipython notebook &
Or after a "normal" run, hit [Control+z] and run the bg command
(some lecture).
you can convert .ipynb file into .py file using nbconvert
with Ipython notebook 2.x (some lecture):
ipython nbconvert --to python mynotebook.ipynb
with Ipython notebook 3.x (some lecture):
ipython nbconvert --to script mynotebook.ipynb
.ipynb contains your script AND results AND formatted text.
You can compare it to a literate script (but using "machine programming language").