I have a jupyter notebook that creates a local cache of images. I would like to clear the cache before I run a particular cell. Does anyone know how to do that? I tried:
import gc
gc.collect()
But it had no effect. I also tried clear all the cookies and cache in chrome, but that did work either.
Try the following in a new cell of your Jupyter Notebook:
%reset -f
The above will clear all variables without User Confirmation. If you want explicit user confirmation, just drop the -f.
%reset_selective -f varName
The above will delete particular variable.
In case the Python version you are using is greater than 3.3, you may also try (I haven't used it!):
%store -d someVar # Remove particular variable and its value from storage
%store -z # Remove all variables from storage
For more info, check this doc.
I realise this isn't a great (bordering on terrible) answer - in short, close and re-open the browser.
I've been struggling with this myself today, and even clearing the browser cache did not seem to work. The only way I've found refresh images is to:
Save your notebook
Halt and close the session
Close the browser
When you re-open the notebook - the new images will appear
... I even tried deleting the referenced image file. But the browser still held onto it.
This worked for me, when an old image would not be refreshed in a markdown cell in Jupyter-Lab on Chrome:
Give the image path a bad path in your markdown code
Execute that markdown cell so it errors.
Rename the image path correctly.
Voila! It should refresh.
Python in jupyter is not the same, i think that the automatic garbage collection will work at some point in the code, every step is paused and cached, this makes it very memory intensive to deal with buffers.
The simples solution es to add a destructor to the elements and remove de need for a gc within the requires steps.
This example uses empty buffers as a load, if you have enough ram to open all instances at the same time it will be enough to just use del img_buffs if not handle them in chunks.
n_images = 1024
img_buffs = {}
for i in range(n_images):
img_buffs[i] = (0,) * 150000
for i in range(n_images):
del img_buffs[i]
The best and easiest solution is to first uninstall the Jupyter notebook then re-install it.
Related
After having searched for a long time for a solution to my problem without succeeding, I'd like to ask you my question here.
I have a python code that creates a geoTIFF file from google earth engine data. I'm running it on jupyter notebook and I want to export the geoTIFF to my google drive.
The code works without error and a shapefile (shp) is embedded as input.
The problem is that nothing appears on my drive, the folder "GEE" that it creates is well created, but it is empty.
Here is the export part of the code:
task = ee.batch.Export.image.toDrive(image=bare1.clip(aoi),
scale=10,
region=aoi.getInfo()['coordinates'],
fileFormat='GeoTIFF',
description='Active',
folder='GEE',
maxPixels=1e9)
task.start()
You should also know that I am a beginner in python :)
Do you have an idea for a solution? Do not hesitate to ask me for more details.
Thanks :)
First: Have you checked the code editor (https://code.earthengine.google.com/) to see if there has been an error message that accounts for the lack of export, or if the file is actually being deposited in a different place? One note about the 'folder' parameter, is that (in my understanding) it doesn't create a folder necessily, but instead is just telling GEE to deposit your image in the most recently created folder of the same name, which could be anywhere in your Drive.
Next, have you definitely mounted your Google Drive? I assume so, if the GEE folder is working, but just to be sure you can always run:
from google.colab import drive
drive.mount('/content/drive')
Next, I have found that when I am exporting an image, I need to convert it to double for correct export. So in your case, this would be changing the first line to (adding the .toDouble())
task = ee.batch.Export.image.toDrive(image=bare1.clip(aoi).toDouble())
If that doesn't work: Have you tried exporting other images with this same code? (Ie replacing bare1 with another image that you know works, like ee.Image(1) which makes a blank image where every pixel is the value 1?
Happy to take another look if none of this helps!
The task console in the GEE code editor should give a description of the export error. Exports are finicky with a number of causes for error. A good first place to check is that you didn't exceed the maximum pixels. You can deal with max pixel errors by reducing the number of bands in your image to only include those that you need, or increasing the maxpixel parameter in your export task. Sometimes the following dictionary style formatting works for me although it's not clear why:
task = ee.batch.Export.image.toDrive(**{
'image':bare1.clip(aoi),
'scale':10,
'region': aoi.getInfo()['coordinates'],
'fileFormat':'GeoTIFF',
'description':'Active',
'folder':'GEE',
'maxPixels':1e9
})
task.start()
I made a function in the jupyter notebook and the cell got deleted. I want to know the code in the function. I cannot use undo deleted cell because it got deleted a long time ago. The function is not deleted from the notebook memory as I can run it. Is there a way to get the code written inside the function?
You might still be able to get the source code of the function, if you still have it in memory like you say.
This post gives a way to try and get it through the inspect library:
https://stackoverflow.com/a/427533/14486144
import inspect
lines = inspect.getsource(foo)
print(lines)
Try give that a shot, I tried it out in jupyter notebooks, although I wasn't exactly sure how to recreate your situation that you can't undo deleted cells. Try that out and see it if works.
I want to know if there is an option in the Notebook UI or some magic command (for a cell) that prevents its re-execution.
For e.g., if I execute a few cells in sequence, and wanted to go back to re-run the first cell, then I should be prevented from doing that.
For the old Jupyter notebook see: How to freeze a cell alone with its outputs on a jupyter notebook.
For JupyterLab there is no extension yet but there is an agreement that this is a good idea. Everyone can become an extension author and contribute it. A good starting point is the Extension Developer guide.
If the cell has no output that you care about you can use a handy trick of temporarily switching its type to "raw" (as raw cells don't get executed). It can be done with a keyboard sequence of Esc, r; then to restore its type to "code" cell you can use Esc, y.
I'm starting to learn Python and I'm currently following a MOOC using Jupyter, which seems to be a great tool. It's composed of several questions, each one requiring to write a piece of code.
I submit my code on a regulary basis so I know that this particular notebook works. But now I have an orange "Method not allowed" flag top right on my browser:
When I click on it it disappears for a while and then come back. All code chunks are executed without any error.
How could I get informations about how to solve this issue ?
EDIT :
It wasn't about the code written in the notebook. See answer below.
Thanks :)
I finally figured out what it was for my case :
This flag meant that some command from Jupyter menu somehow failed/was rejected (e.g. save & checkpoint at a moment my login session had expired)
When using automatic commands (e.g. auto-save), it can appear randomly and be hard to understand
It had no link with code in the notebook
This issue might have been caused by the fact I used Jupyter locally for personal purposes, at the same time I used MOOC's Jupyter.
Reloading the page fixed the issue but I lost some pieces of the code I wrote and I wasn't able to download the file before reloading (it failed too), so beware!
I had the same problem (the orange 'method not allowed' message was appearing, and autosave and save were failing). Seems like it might be caused by an expired session as suggested by kdallaporta.
A solution that avoids reloading (as suggested by kdallaporta) and possible data loss is the following (this worked for me in a coursera notebook): log out and then log in from another window, and then open a new copy of the notebook (without closing the original notebook). The original notebook should work again without the problems above, and the new copy can be closed.
I would like to access the textual contents of another cell in the notebook from Python so that I can feed it to the unit testing script, regardless of whether it has been executed or not. It seems like this should be possible, but I can't find the right API to do it in the IPython kernel. Thoughts?
This question asks the same question, but I don't really want to use magics unless I have to. Ideally the workflow would be "select widget choice" followed by "click widget" to cause the tests to run.
Background
I'm working with some students to teach them Python, and in the past when I've done this I've set up a bunch of unit tests and then provided instructions on how to run the tests via a shell script. However, I'm working with some students that don't have access to computers at home, so I decided to try and use an Jupyter notebook environment (via mybinder.org) to allow them to do the same thing. I've got most of it working already via some ipywidgets and a helper script that runs the unit tests on some arbitrary set of code.
As far as the cells that have been run, the input and output caching system described at https://ipython.readthedocs.io/en/stable/interactive/reference.html#input-caching-system might be useful. (Examples of its use at https://stackoverflow.com/a/27952661/8508004 ). It works in Jupyter notebooks as shown below. (The corresponding notebook can be viewed/accessed here.)
Because #A. Donda raised the issue of markdown in the comments below, I'll add here that nbformat provides related abilities that works with saved notebook files. Reading a saved notebook file with nbformat allowa getting cells and content, no matter if it is code or markdown, and sorting whether the cells are markdown or code, etc.. I have posted a number of examples of using nbformat on the Jupyter Discourse forum that can be seen listed via this search here. It offers more utility than the related json.load(open('test.ipynb','r')) command, highlighted in a comment here to read a notebook file because of the additional notebook context automatically included.
If you want to capture the contents of a specific cell to access it from another one, one workaround seems to bee to write the contents of the first cell to file when executing it and later loading the resulting file (i.e. the text giving the former cell's content) inside the latter cell, where the content of the former cell is required.
Cell's contents can be saved to file (when executing the respective cell) via the command
%%writefile foo.py
which has to be placed at the beginning of a cell. This results in the cell's content (in which the upper command is executed) being saved to the file foo.py and it's just a matter of later reading it in, again.
The output of a cell can be made available more easily:
Just place %%capture output in the first line of a cell. Then, the output of the cell (after execution) is going to be saved as a string to the variable output and can be used like any standard python string-variable.
References:
Programmatically get current Ipython notebook cell output? and
https://nbviewer.jupyter.org/github/ipython/ipython/blob/1.x/examples/notebooks/Cell%20Magics.ipynb
I found a solution to this that can get the output of any cell. It requires running some Javascript.
Here is the code that you can put in a cell and run it an it will generate a Python variable called cell_outputs which will be an array of the cell outputs in the same order as they appear on the page. The output of the cell executing this code, will have an empty string.
%%js
{
let outputs=[...document.querySelectorAll(".cell")].map(
cell=> {
let output=cell.querySelector(".output_text")
if(output) return output.innerText
output=cell.querySelector(".rendered_html")
if(output) return output.innerHTML
return ""
}
)
IPython.notebook.kernel.execute("cell_outputs="+JSON.stringify(outputs))
}
If you need the output of a specific cell, just use its index, eg: cell_outputs[2] in your Python code to access the output of cell #3.
I tested this on my Jupyter notebook 6.0.3 (recent install via the Anaconda community edition) on Google Chrome. The above could should work fine on any modern browser (Chrome, Firefox or Edge).