Dear all I am running the first Azure tutorial for the MNIST dataset.
It says that the utils.py should be in the same folder as the code. I tried to install python-utils in myconda-environment, but that did not solve the problem. After using pip install utils I rather made it worse :-(
It is probably simple but I am stuck.
How would you do that on the notebook running on:
locally
in a Azure notebook
I use Anaconda with a separate environment running the Azure SDK and python 3.6.
According to your description, I think the first Azure tutorial for the MNIST dataset is Tutorial: Train an image classification model with Azure Machine Learning service.
You can find all of source codes via the link inside the tutorial as below at here.
Get the notebook
For your convenience, this tutorial is available as a Jupyter notebook. Run the tutorials/img-classification-part1-training.ipynb notebook either in Azure Notebooks or in your own Jupyter notebook server.
Here is the source code of utils.py.
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
import gzip
import numpy as np
import struct
# load compressed MNIST gz files and return numpy arrays
def load_data(filename, label=False):
with gzip.open(filename) as gz:
struct.unpack('I', gz.read(4))
n_items = struct.unpack('>I', gz.read(4))
if not label:
n_rows = struct.unpack('>I', gz.read(4))[0]
n_cols = struct.unpack('>I', gz.read(4))[0]
res = np.frombuffer(gz.read(n_items[0] * n_rows * n_cols), dtype=np.uint8)
res = res.reshape(n_items[0], n_rows * n_cols)
else:
res = np.frombuffer(gz.read(n_items[0]), dtype=np.uint8)
res = res.reshape(n_items[0], 1)
return res
# one-hot encode a 1-D array
def one_hot_encode(array, num_of_classes):
return np.eye(num_of_classes)[array.reshape(-1)]
If you want to import it in an Azure Jupyter Notebook, please see my steps below.
Move into your project page, and click the New buttom and select the Blank File.
Then name the file utils.py and press Enter key.
Select the file and click Edit File.
Copy and paste the content of utils.py from the tutorial Github repo, and click Save File.
Create a Notebook to test import utils, it works.
So # make sure utils.py is in the same directory as this code means as the figure below.
Related
so I am still new to python and I tried to use a script that create bokeh effect on an image written by someone else. Here is the repository : Bokeh-Effect-in-Image-using-Python
Here is more information about how the code works from the author: bokeh-effect-in-image-using-python
Notice that I am coding on VSCode, macOS BigSur, Python 3.9.
First I created a folder for my project then I created a virtual environment, it has been recognised by VsCode and incorporated into the project (so I have the .vscode folder with the json file), I created a python file in which I pasted the code and I downloaded the packages I needed inside the virtual environment.
I also tried to add different lines in the json file based on what I read on the web but it doesn't work.
The problem is that the code runs until it encounters lines using OpenCV package, I don't have any Traceback but the code just do nothing.
Here is what I have in the terminal.
Terminal
I tried to use print at different places and to remove certain lines which use opencv and the problem occurred each time opencv was used or when Tracker_HSV was used (if I deleted all the lines using opencv before) but I think it is because Tracker_HSV use OpenCV.
So maybe I made a mistake during the setup of the project or opencv is not working correctly, I don't know what to do.
(Sorry for my English, I am French.)
According to the information you provided, for the use of the module "opencv", it is recommended that you try the following:
Please install the module "opencv" correctly: "pip install opencv-python"
Please use the absolute path of the picture in the code here: "img = cv.imread("D:/.../Bokeh_Demo/flower.jpg")"
When importing the module "cv2", because it contains other files named "cv2", (so pylint reported some errors here, but this does not affect the execution of the code). We can use "from cv2 import cv2 as cv" to distinguish them.
Run:
Update 1：
Please check that the module "opencv" has been successfully installed in your currently selected VSCode environment：
More reference: Environment in VSCode.
Update 2:
# import cv2
from cv2 import cv2 as cv
import numpy as np
import Tracker_HSV as tr
from PIL import Image
import os
# cv2.namedWindow('mask', cv2.WINDOW_NORMAL)
cv.namedWindow('mask',cv.WINDOW_NORMAL)
cv.resizeWindow('mask',(500,500))
filename= 'flower.jpg'
cwd = os.getcwd()
name_file=os.path.splitext(filename)[0]
path_save_temp=os.path.join(cwd,'Data')
path_save_folder=os.path.join(path_save_temp,f'{name_file}_blur_data')
if not os.path.exists(path_save_folder):
os.makedirs(path_save_folder)
img = cv.imread("D:/....../test_Bokeh/Bokeh_Demo/flower.jpg")
blur = cv.GaussianBlur(img,(5,5),0)
img_hsv=cv.cvtColor(img,cv.COLOR_BGR2HSV)
file_save_blur= os.path.join(path_save_folder,'blur.png')
im_blur = cv.GaussianBlur(img,(81,81),0)
cv.imwrite(file_save_blur,im_blur)
xs,ys,w,h = cv.selectROI('mask',img)
crop_img=crop_img_true=crop_img_contour=img[ys:ys+h, xs:xs+w]
if not crop_img_true.shape[0]> 1:
crop_img_true=img
x,y,z,a,b,c=(tr.tracker(crop_img_true))
crop_img_true=cv.cvtColor(crop_img_true,cv.COLOR_BGR2HSV)
file_save_mask_inrange= os.path.join(path_save_folder,'mask inRange.png')
mask_inRange=cv.inRange(crop_img_true,(x,y,z),(a,b,c))
cv.imwrite(file_save_mask_inrange,mask_inRange)
_, threshold = cv.threshold(mask_inRange, 250, 255, cv.THRESH_BINARY)
Gauss_threshold =cv.adaptiveThreshold(threshold,255,cv.ADAPTIVE_THRESH_GAUSSIAN_C,cv.THRESH_BINARY_INV,101,10)
blank_space_black= np.zeros((crop_img_true.shape[0],crop_img_true.shape[1]),np.uint8)
blank_space_black[:]=(0)
_,contours,_ = cv.findContours(Gauss_threshold, cv.RETR_TREE, cv.CHAIN_APPROX_NONE)
maxi=cv.contourArea(contours[0])
c=[]
for cnt in contours:
if cv.contourArea(cnt)>=maxi:
maxi=cv.contourArea(cnt)
## print(cv2.contourArea(cnt))
c= cnt
file_save_contour= os.path.join(path_save_folder,'Contour.png')
cv.drawContours(crop_img_contour, c, -1, (0, 255, 0), 5)
cv.imwrite(file_save_contour,crop_img_contour)
file_save_poly= os.path.join(path_save_folder,'mask fill poly.png')
mask_poly=cv.fillConvexPoly(blank_space_black,c,(255,255,255))
cv.imwrite(file_save_poly,mask_poly)
crop_img_true=cv.cvtColor(crop_img_true,cv.COLOR_HSV2BGR)
file_save_mask_bitwise= os.path.join(path_save_folder,'mask bitwise and.png')
mask_bitwise_and = cv.bitwise_and(crop_img_true,crop_img_true,mask=mask_poly)
cv.imwrite(file_save_mask_bitwise,mask_bitwise_and)
im2= Image.open(file_save_mask_bitwise)
im2=im2.convert('RGBA')
datas=im2.getdata()
newdata=[]
for data in datas:
if data[0]== 0 and data[1]== 0 and data[2]== 0:
newdata.append((255,255,255,0))
else:
newdata.append(data)
file_save_transparent= os.path.join(path_save_folder,'transparent.png')
im2.putdata(newdata)
im2.save(file_save_transparent)
im_blur= Image.open(file_save_blur)
file_save_final= os.path.join(path_save_folder,'final.png')
im_blur.paste(im2,(xs,ys),im2)
im_blur.save(file_save_final)
im_final= Image.open(file_save_final)
im_final.show('Final Result')
cv.imshow('GaussianBlur',blur)
cv.waitKey(0)
cv.destroyAllWindows()
After exporting a jupyter-gmaps base map and directions to HTML using "ipywidgets.embed.embed_minimal_html" I'm having trouble showing the directions layers when I re-open the HTML map.
The base map and directions layers show up just fine in jupyter notebook as shown here:
Map and directions rendered in Jupyter Notebook
However, when the HTML file is re-opened using either a server (opened using "python -m http.server 8080"), or by double clicking on the saved file, or by using the code snippet below, only the base layer renders as shown here, along with some strange artifacts: What is rendered when the exported HTML is opened
import IPython
IPython.display.HTML(filename='testest.html')
The directions layers seem to be saving in the HTML because I get an error regarding the directions layers when I try to save the image, but the scripts aren't being rendered in the browser (Error about directions layer when opening the HTML file). Also, the following errors are displayed Chrome console.
Here is the code I used to generate this map, if someone could help me figure out how to get the directions layers to render in the exported map I would really appreciate it :) .
import gmaps
from ipywidgets.embed import embed_minimal_html,embed_data
import IPython
import os
#configure api
api_key = 'YOUR-API-KEY-HERE'
gmaps.configure(api_key=api_key)
#Define location 1, 2, and 3
Durango = (37.2753,-107.880067)
meow = (37.3654,-106.880067)
SF = (37.7749,-122.419416)
#Create the map
fig = gmaps.figure()
#create the layers
layer1 = gmaps.directions_layer(Durango, meow,travel_mode="DRIVING")
fig.add_layer(layer1)
layer2 = gmaps.directions_layer(Durango, SF,travel_mode="DRIVING")
fig.add_layer(layer2)
#Export the map and layers to HTML
embed_minimal_html(os.getcwd() + '\{}.html'.format("testest"), views=[fig],title='test')
#show the map in Jupyter
fig
Thanks!!
I'm trying to load the English model for StanfordNLP (python) from my local machine, but am unable to find the proper import statements to do so. What commands can be used? Is there a pip installation available to load the english model?
I have tried using the download command to do so, however my machine requires all files to be added locally. I downloaded the english jar files from https://stanfordnlp.github.io/CoreNLP/ but am unsure if I need both the English and the English KBP version.
directory set for model download is /home/sf
pip install stanfordnlp # install stanfordnlp
import stanfordnlp
stanfordnlp.download("en") # here after 'Y' one set custom directory path
local_dir_store_model = "/home/sf"
english_model_dir = "/home/sf/en_ewt_models"
tokienizer_en_pt_file = "/home/sf/en_ewt_models/en_ewt_tokenizer.pt"
nlp = stanfordnlp.Pipeline(models_dir=local_dir_store_model,processors = 'tokenize,mwt,lemma,pos')
doc = nlp("""One of the most wonderful things in life is to wake up and enjoy a cuddle with somebody; unless you are in prison"""")
doc.sentences[0].print_tokens()
I am unclear what you want to do.
If you want to run the all-Python pipeline, you can download the files and run them in Python code by specifying the paths for each annotator as in this example.
import stanfordnlp
config = {
'processors': 'tokenize,mwt,pos,lemma,depparse', # Comma-separated list of processors to use
'lang': 'fr', # Language code for the language to build the Pipeline in
'tokenize_model_path': './fr_gsd_models/fr_gsd_tokenizer.pt', # Processor-specific arguments are set with keys "{processor_name}_{argument_name}"
'mwt_model_path': './fr_gsd_models/fr_gsd_mwt_expander.pt',
'pos_model_path': './fr_gsd_models/fr_gsd_tagger.pt',
'pos_pretrain_path': './fr_gsd_models/fr_gsd.pretrain.pt',
'lemma_model_path': './fr_gsd_models/fr_gsd_lemmatizer.pt',
'depparse_model_path': './fr_gsd_models/fr_gsd_parser.pt',
'depparse_pretrain_path': './fr_gsd_models/fr_gsd.pretrain.pt'
}
nlp = stanfordnlp.Pipeline(**config) # Initialize the pipeline using a configuration dict
doc = nlp("Van Gogh grandit au sein d'une famille de l'ancienne bourgeoisie.") # Run the pipeline on input text
doc.sentences[0].print_tokens()
If you want to run the Java server with the Python interface, you need to download the Java jar files and start the server. Full info here: https://stanfordnlp.github.io/CoreNLP/corenlp-server.html
Then you can access the server with the Python interface. Full info here: https://stanfordnlp.github.io/stanfordnlp/corenlp_client.html
But just to be clear, the jar files should not be used with the pure Python pipeline. Those are for running the Java server.
I'm new to Machine Learning and I'm following a Sentdex tutorial on Google Colab. It's supposed to be a ML program that distinguishes between cat and dog images. However, whenever I run my code, somethings wrong with my 'file or directory.'
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\atlgwc16\\PetImages/Dog'
I honestly don't know where Google Colab stores its files so I don't know where to put the folder of images.
Here is my full code so far:
import numpy as np
import matplotlib.pyplot as plt
import os
import cv2
from tqdm import tqdm
DATADIR = "C:\Users\atlgwc16\PetImages"
CATEGORIES = ["Dog", "Cat"]
for category in CATEGORIES:
path = os.path.join(DATADIR, category)
for img in os.listdir(path):
img_array = cv2.imread(os.path.join(path,img), cv2.IMREAD_GRAYSCALE)
plt.imshow(img_array, cmap = 'gray')
plt.show()
break
Tutorial being followed as referenced in the question:
https://pythonprogramming.net/loading-custom-data-deep-learning-python-tensorflow-keras/
Since you are using Google Colab, you can upload the Kaggle dataset of dog and cat images to Google Drive. See the Google Colab Jupyter notebook provided by Google that explains how to do this:
https://colab.research.google.com/notebooks/io.ipynb#scrollTo=u22w3BFiOveA
You would then access files from your Google Drive (in this case, the training set after you upload it to Google Drive) much in the same way as accessing files locally on your computer.
This is the example provided in the link above:
with open('/content/gdrive/My Drive/foo.txt', 'w') as f:
f.write('Hello Google Drive!')
!cat /content/gdrive/My\ Drive/foo.txt
So, since you are using Google Colab, you would need to adjust the code from the Sentdex tutorial to work better with the notebook you are creating. Google Colab uses Jupyter notebooks. Each cell in the notebook runs off of the same 'session'. So, if you import a Python module in one cell, it can be used in the next cells. It's magic like that.
It would look like this:
[CELL 1]
from google.colab import drive
drive.mount('/content/gdrive')
You will then give permission for Google Colab to access your Google Drive.
[CELL 2]
import numpy as np
import matplotlib.pyplot as plt
import os
import cv2
from tqdm import tqdm
DATADIR = '/content/gdrive/My Drive/PetImages/'
#^See?#
# You would need to go to Google Drive and create the 'PetImages' folder at the top level of your Google Drive. You would upload the data set to the PetImages folder creating a 'Dog' subfolder and a 'Cat' subfolder.
CATEGORIES = ["Dog", "Cat"]
for category in CATEGORIES: # do dogs and cats
path = os.path.join(DATADIR,category) # create path to dogs and cats
for img in os.listdir(path): # iterate over each image per dogs and cats
img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE) # convert to array
plt.imshow(img_array, cmap='gray') # graph it
plt.show() # display!
break # we just want one for now so break
break #...and one more!
After properly uploading the data set to Google Drive and using the special google.colab module, you should be able to easily access your training data. Google Colab is a cloud-based tool for creating Jupyter notebooks and running Python programs. So, while similar to running a Python program locally on your computer, it is not exactly the same. It would help to read through how Google Colab works more if you want to use it completely in the cloud--using GDrive to store files rather than your own computer. See the link I posted above from Google.
Happy coding.
I did it for my self and it works for me.
I use data set from my local drive such as a hard disk.
note: your dataset folder must be in the zip form.
first, follow the method with me and you will access your dataset from the local drive.I use google colab. first, create a Jupyter notebook in google Colab and run the below code step by step:
first step: run the below code in your notebook and upload your dataset from your hard drive or local drive
from google.Colab import files
uploaded = files.upload()
when the process is complete 100 percent and do the second step:
second step:
copy and run the below code, this step will unzip the dataset
import zipfile
import io
zf = zipfile.ZipFile(io.BytesIO(uploaded['DogVsCat.zip']), "r")
zf.extractall()
third step: run the code it will import all the required libraries
import numpy as np
import os
import cv2
import matplotlib.pyplot as plt
this will import all the required libraries for you.
fourth step:
specify the path for specifying the path do below steps:
fist: check the image for more ease of your
on the left corner folder icon click on the highlighted folder and you will see your unzip dataset in my case my dataset is "DogVsCat",
note: there you will see two kinds of dataset zip and unzip, you copy the path of unzip data.
right click on it and copy the path from it and
run the below code:
DIRECTORY ='/content/DogVsCats'
CATEGORIES = ['cats', 'dogs']
note: please add your path in DIRECTORY(This directory is the path for me) path, not my path. and again run the code:
note: please add your own folder names in CATEGORIES not my folder names for more information see the picture:
my dataset structure
at the end create train data
5th step:
data = []
for category in CATEGORIES:
path = os.path.join(DIRECTORY, category)
for img in os.listdir(path):
img_path = os.path.join(path, img)
label = CATEGORIES.index(category)
arr = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)
new_arr = cv2.resize(arr, (60, 60))
data.append([new_arr, label])
sixth step:
print the data:
run the below code to show you
data
seventh step: shuffle your data:
import random
random.shuffle(data)
eight-step:
specify features and labels for training the model
X = []
y = []
for features, label in data:
X.append(features)
y.append(label)
X = np.array(X)
y = np.array(y)
ninth-step: print features
X
tenth-step: print labels
y
note I can not share all the code with you for the lack of time.
note: for more clearness check my code pictures:
pic1-Of-My-Code
pic2-of-my-code
I'm trying to read a jpg file using Pillow (Version 3.2.0) in Jupyter notebook (Python 3.4), but it fails with the following error:
OSError: broken data stream when reading image file
I'm using the following code:
from PIL import Image
im = Image.open("/path/to/image.jpeg")
im.show()
It works fine both in the interactive Python shell and using Python 2.7 instead of 3.4.
I've followed these steps already: Using Pillow with Python 3
Anyone an idea what's going on?
Looks like you're not pointing to the directory where your photo is stored.
import os
defaultWd = os.getcwd()
defaultWd # Sets your curretn wd
os.chdir(defaultWd + '\\Desktop') # Points to your photo--e.g., on Desktop
os.getcwd() # Shows change in wd
from PIL import Image
im = Image.open("Mew.jpg")
im.show() # Will plot to your default image viewing software
And another way if you don't want to change current wd:
im = Image.open(os.getcwd() + "\\Desktop\\Mew.jpg")
im.show()
And if you want to plot inline:
from matplotlib.pyplot import imshow
%matplotlib inline
inlinePic = Image.open(os.getcwd() + "\\Desktop\\Mew.jpg")
imshow(inlinePic)
Note: You may also want to simply try typing 'jpg' instead of 'jpeg' as you did above, if your image is in your current working directory. Also, if PIC is not installed, you'll get this error NameError: name 'Image' is not defined.
The problem was related to another import: I was importing Tensorflow before PIL, which caused the problem. Same issue as this one: https://github.com/scikit-image/scikit-image/issues/2000. Changing the order of the imports solved it.