I have a dataset of images on Dropbox it is in uncompressed folders, and I want to train a a model using google colab. How can I tell Colab where the dataset is in dropbox
Related
I've seen tons of tutorials about using YOLOv7 for object detection, and I followed one to detect license plates. I cloned the repo from github, put my data in, uploaded to drive and ran it on google collab. The tutorials stop at testing. How do I save this model and use it later for predictions? Is it already saved in the uploaded folder, and all I have to do is put inputs into the test folder and run the below line? Or can I save this as a .yaml file on my local drive? If so, can you share the code for using that .yaml file to make a prediction locally? (Btw the training used the collab GPU)
This is the line I'm using for testing:
!python detect.py --weights best.pt --conf 0.5 --img-size 640 --source img.jpg --view-img --no-trace
As written in the question, I would like to download a csv dataset from the website ourworldindata.org for further data manipulation using Google Colab. In principle, I could download it to my machine and upload it to Colab or save it to my Google Drive after having linked it to the Colab worksheet.
However, I was wondering if there is a more straightforward way to get the data.
You can download datasets using !wget 'link to the file', but in ourworldindata.org, you don't get a specific link to download the .csv file is what I observed in most of the cases.
But for a few datasets, you can find the link to download the .csv file and you can use
cd 'working_folder'
!wget https://covid.ourworldindata.org/data/owid-covid-data.csv
to download the file directly to your working folder
So I have my images links like:
https://my_website_name.s3.ap-south-1.amazonaws.com/XYZ/image_id/crop_image.png
and I have almost 10M images which I want to use for Deep Learning purpose. I have a script to download the images and save them in desired directories already using requests and PIL
Most naïve idea that I have and which I have been using my whole life is to first download all the images in my local machine, make a zip and upload it to Google Drive where I can just use gdown to download it anywhere based on my Network Speed. Or just copy to Colab using terminal.
But that data was not so big. Always under 200K images. But now, the data is huge so downloading and again uploading the images will take a whole lot of time in days and on top of that, it'll just make the Google Drive run out of space with 10M images. So I am thinking about using AWS ML (SageMaker) or something else from AWS. So is there a better approach to this? How can I import the data directly to my SSD supported based virtual machine?
You can use the AWS python library boto3 to connect to the S3 bucket from Colab: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html
I would like to pay for Google Colab Pro in order to have full access to the GPus, however my google drive is already full, so does that mean that, besides paying the Google Colab, should I also pay for the Google drive space extension? my drive is already full and I have a 30GB dataset to classify in Google Colab. So, is there any Python trick to load and manipulate my dataset in Colab without having to read the dataset from google drive? Can I load the dataset from my computer or, say, from another cloud service like Dropbox?
P.s: there is a related question here, but the solution seems to not work.
I want to train a deep learning (CNN) model on a dataset containing around 100000 images. Since the dataset is huge (approx 82 Gb), I want to use Google colab since it's GPU supported. How do I upload this full image folder into my notebook and use it?
I can not use google drive or git hub since my dataset is too large.
You can try zipping and the unzip on collab
Step 1: Zip the whole folder
Step 2:Upload the folder
Step 3: !unzip myfoldername.zip
Step 4:type ls and see folder names to see if successful
It would be better if you compress or resize the images to reduce file size using opencv or something