Im trying to download and read a geojson file from url to use it latter to create a folium map, I already install wget on mac using brew.
when running the code I get this
# Download and store a geojson file of Indiana containig AGEB boundaries
import wget
import geojson
!wget https://github.com/Alexrendon/Indianapolis-data/blob/main/Indiana_censustracts.geojson
census_tract = r'Indiana_censustracts.geojson'
print("geojson ready!")
OUTPUT
zsh:1: command not found: wget
Solution was to use !curl
import wget
import geojson
!curl -O https://github.com/Alexrendon/Indianapolis-data/blob/main/Indiana_censustracts.geojson
census_tract = r'Indiana_censustracts.geojson'
print("geojson ready!")
Your code
# Download and store a geojson file of Indiana containig AGEB boundaries
import wget
import geojson
!wget https://github.com/Alexrendon/Indianapolis-data/blob/main/Indiana_censustracts.geojson
census_tract = r'Indiana_censustracts.geojson'
print("geojson ready!")
looks like you are confusing wget python module available at PyPI and GNU Wget command line tool. If you want just to downlad file, neither is required as there exist urlretrieve inside urllib.request which is part of python standard library. Consider following simple example
import urllib.request
urllib.request.urlretrieve("https://www.example.com","example.html")
first argument is URL, second is filename
The error tells you that you don't have wget on your machine. To install it on Mac, just do
brew install wget
Related
In a jupyter notebook running on anaconda there is a line "import utils.lib as lib". When I run it, I get the error message "ModuleNotFoundError: No module named 'utils.lib'".
I tried to search for the utils.lib on Internet so that I can install it. But I could not find it. Please let me know how to install it. Thank you -- Manoranjan Dash
You don't provide the code context for the import line. You don't have to always provide the complete context; however, you didn't provide anything and that is why the Bot tried to encourage you to improve things. 'Additional context' would also be important to supply in your post such as the source of the notebook or any related blog post, etc..
The only place I see that import come up on the internet besides your post is here. The code there shows how to get that and install it under 'Downloading the utils and installing':
LIB_DIRECTORY_PATH = DIR+'/utils'
# Check if utils directory already exist, otherwise download, and install
import os
import shutil
if not os.path.isdir(LIB_DIRECTORY_PATH):
if not os.path.isdir(DIR+'/utils'):
os.mkdir(DIR+'/utils')
print('Downloading utils')
user = "ruslanmv"
repo = "Speech-Recognition-with-RNN-Neural-Networks"
src_dir = "utils"
pyfile = "lib.py"
url = f"https://raw.githubusercontent.com/{user}/{repo}/master/{src_dir}/{pyfile}"
!wget --no-cache --backups=1 {url}
print("Installing library...")
shutil.move(DIR+'/lib.py', DIR +'/utils/lib.py')
print("Done.")
Source of that above code: https://ruslanmv.com/blog/Speech-Recognition-with-RNN-Neural-Networks
Code that the code block retrieves and places in the correct location is found at this repo. It isn't something you install. You need to place it alongside the notebok. (Ideally it is set up so you just download or clone the repository, it looks like to me.) The utils directory and it's content is what you need to get or make/copy and place along with your notebook.
Direct link to raw code it gets:
https://raw.githubusercontent.com/ruslanmv/Speech-Recognition-with-RNN-Neural-Networks/master/utils/lib.py
I'm trying to export an image I've created with plotly as a .svg file.
This is the code I'm running to export it:
fig.write_image("/content/fig1.svg")
This is the error I'm getting:
I've entered in the github page of Orca and tried to use the installation codes they suggest but neither one worked.
You can install orca with the following code. (a library I wrote)
!pip install kora -q
import kora.install.orca
Then it can write_image
import plotly.graph_objects as go
fig = go.Figure( go.Scatter(x=[1,2,3], y=[1,3,2] ) )
fig.write_image("image.png")
from IPython.display import Image
Image("image.png")
If you prefer to install orca yourself
import os
from urllib.request import urlretrieve
url = "https://github.com/plotly/orca/releases/download/v1.2.1/orca-1.2.1-x86_64.AppImage"
orca = '/usr/local/bin/orca'
urlretrieve(url, orca)
os.chmod(orca, 0o755)
os.system("apt install xvfb libgconf-2-4")
I am trying tensorflow in Google Colaboratory from this codelab,
I need to download 'http://download.tensorflow.org/example_images/flower_photos.tgz' this file to complete the lab.
How I can download the file. Is there anyway to upload the tar file without downloading it on my machine.
I tried this method
import urllib
testfile = urllib.URLopener()
testfile.retrieve("http://randomsite.com/file.gz", "file.gz")
This doesn't work, wget is not found also. Anyone please tell me how to do this.
Possibly simpler: use !wget, e.g., executing a cell with the command:
!wget https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png
will save the file to the local filesystem of the VM. (Note the leading !. This is a signal to execute the line as a shell command.)
Read the manual, they have good examples and explanations: urllib.request
To download:
>>> import os
>>> import urllib.request
>>> urllib.request.urlretrieve('https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png', 'google.png')
('google.png', <http.client.HTTPMessage object at 0x7fba3c4cb908>)
>>> os.listdir()
['google.png']
To just check the content:
>>> with urllib.request.urlopen('http://www.python.org/') as f:
... print(f.read(300))
I am trying to upload a python lambda function with zipped dependencies but for some reason I am constantly getting
"errorMessage": "Unable to import module 'CreateThumbnail'"
whenever I test it.
Here are the steps I took which were almost identical to these docs.
Created and activate a virtualenv with virtualenv ~/lambda_env and source ~/lambda_env/bin/activate
Install Pillow and boto3 with pip install Pillow and pip install boto3
Zip dependencies with cd $VIRTUAL_ENV/lib/python2.7/site-packages and zip -r9 ~/CreateThumbnail.zip *
Add the actual python lambda function to the zip file with zip -g ~/CreateThumbnail.zip CreateThumbnail.py where CreateThumbnail.py is
from __future__ import print_function
import boto3
import os
import sys
import uuid
from PIL import Image
import PIL.Image
s3_client = boto3.client('s3')
def resize_image(image_path, resized_path):
with Image.open(image_path) as image:
image.thumbnail(tuple(x / 2 for x in image.size))
image.save(resized_path)
def handler(event, context):
for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
upload_path = '/tmp/resized-{}'.format(key)
s3_client.download_file(bucket, key, download_path)
resize_image(download_path, upload_path)
s3_client.upload_file(upload_path, '{}resized'.format(bucket), key)
Then in the console I set the handler to be CreateThumbnail.handler
Then I upload CreateThumbnail.zip via the aws console and click 'save & test' I get
"errorMessage": "Unable to import module 'CreateThumbnail'"
I am very confused by this because feel like I am following the docs. Can anyone tell me what I am doing wrong here?
Perhaps check out the lambda-uploader project... It handles the packaging of dependencies and is config based.
https://github.com/rackerlabs/lambda-uploader/
Also these links may be helpful:
http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
https://markn.ca/2015/10/python-extension-modules-in-aws-lambda/
http://www.perrygeo.com/running-python-with-compiled-code-on-aws-lambda.html
The problem lies in the packaging hierarchy. After you install the dependencies, zip the lambda function as follows (in the example below, lambda_function is the name of my function)
Try this:
pip install requests -t .
zip -r9 lambda_function.zip .
zip -g lambda_function.zip lambda_function.py
Do not let your browser automatically unzip the lambda "project" file after downloading. This seems to corrupt the file when it is re-zipped and used.
The tutorial you pointed out uses python 3.8
And you seem to be using python 2.7
That may be the reason.
I am doing a similar tutorial, but they give us the zip ready to upload but warning to select python 3.7 and not 3.8 or it will fail to run correctly.
I need to do a rest-call within a python script, that runs once per day.
I can't pack the "requests" package into my python-package using the AWS Lambdas. I get the error: "Unable to import module 'lambda_function': No module named lambda_function"
I broke it down to the hello_world predefined script. I can pack it into a zip and upload it. Everything works. As soon as I put "import requests" into the file, I get this error.
Here is what I already did:
The permissions of the zip and the project folder (including subfolders) are set to `chmod 777`. So permissions shouldn't be a problem.
The script itself is within the root folder. When you open the zip file, you directly see it.
I installed the requests package into the root-folder of the project using `sudo pip install requests -t PATH_TO_ROOT_FOLDER`
The naming of everything looks like this:
zip-file: lambda_function.zip
py-file: lambda_function.py
handler method: lambda_handler(event, context)
handler-definition in the "webconfig: lambda_function.lambda_handler
The file I want to run in the end looks like this:
import requests
import json
def lambda_handler(event, context):
url = 'xxx.elasticbeanstalk.com/users/login'
headers = {"content-type": "application/json", "Authorization": "Basic Zxxxxxxxxx3NjxxZxxxxzcw==" }
response = requests.put(url, headers=headers, verify=False)
return 'hello lambda_handler'
I'm glad for ANY kind of help. I already used multiple hours on this issue.
EDIT: On Oct-21-2019 Botocore removed the vendored version of requests: https://github.com/boto/botocore/pull/1829.
EDIT 2: (March 10, 2020): The deprecation date for the Lambda service to bundle the requests module in the AWS SDK is now January 30, 2021. https://aws.amazon.com/blogs/compute/upcoming-changes-to-the-python-sdk-in-aws-lambda/
EDIT 3: (Nov 22, 2022): AWS cancelled the deprecation so you can continue to use requests as described below. AWS Blog
To use requests module, you can simply import requests from botocore.vendored. For example:
from botocore.vendored import requests
def lambda_handler(event, context):
response = requests.get("https://httpbin.org/get", timeout=10)
print(response.json())
you can see this gist to know more modules that can be imported directly in AWS lambda.
If you're working with Python on AWS Lambda, and need to use requests, you better use urllib3, it is currently supported on AWS Lambda and you can import it directly, check the example on urllib3 site.
import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'http://httpbin.org/robots.txt')
r.data
# b'User-agent: *\nDisallow: /deny\n'
r.status
# 200
I finally solved the problem: The structure in my zip file was broken. It is important that the python script and the packed dependencies (as folders) are in the root of the zip file. This solved my problem.
It's a bit depressing if you find such easy errors after hours of try and failure.
I believe you have lambda_function.py on the Lambda console. You need to first create the Lambda function deployment package, and then use the console to upload the package.
You create a directory, for example project-dir on your system (locally)
create lambda_function.py in project-dir, copy the content of lambda_function.py from lambda console and paste it in project-dir/lambda_function.py
pip install requests -t /path/to/project-dir
Zip the content of the project-dir directory, which is your deployment package (Zip the directory content, not the directory)
Go to the Lambda console, select upload zip file in code entry type and upload your deployment package. Import requests should work without any error.
With this command download the folder package
pip install requests -t .
Run this command on your local machine, then zip your working directory, then upload to aws.
Most of the comments somehow correct, but not enough informative for AWS beginners. Here is my long resume what needs to be done for accessing requests functionality:
1. Creates root folder for AWS Lambda function
% mkdir lambda-function
2. Go inside crated root folder
% cd lambda-function
3. Create entry point Python file for AWS Lambda.
% vi lambda_function.py
4. Paste a code into lambda_function.py
import requests
def lambda_handler(event, context):
response = requests.get("https://www.test.com/")
print(response.text)
return response.text
5. Install requests library. Note:package folder created
% pip install --target ./package requests
6. Go inside package
% cd package
7. Zip package
zip -r ../deployment-package.zip .
8. Go into parent folder
% cd ..
9. Zip deployment packge and lambda function file
% zip -g deployment-package.zip lambda_function.py
In the AWS Lambda functions tap "Upload from" and pick ".zip file". Navigate to your zip package zip file: deployment-package.zip.
After upload all files will be inside AWS Lambda function.
python 3.8 windows 10
lambda is looking for a specific folder structure and we are going to recreate in this manner in the steps below (https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-create):
make a folder on your desktop called "python," open a cmd terminal: cd desktop
pip install --target python requests
right click your python folder and zip it and rename the zip to 'requests.zip' - now if you look inside the zip you should see the python folder.
aws console > lambda > layers > create layer => name layer/upload requests.zip
aws console > functions > create function => in the "designer" box select layers and then "add layers." Choose custom layers and select
your layer.
Go back to the function screen by clicking on the lambda symbol in the designer box. Now you can see "function code" again. Click lambda_function.py
Now you can import requests like this:
import json
import requests
def lambda_handler(event, context):
# TODO implement
response = requests.get('your_URL')
return {
'statusCode': 200,
'body': json.dumps(response.json())
}
Copy whatever you have in the lambda_function fron AWS lambda console and paste it in a new python script and save it as lambda_function.py.
Make a new folder (I name it as package) and save requests module in it by running the following code in terminal: pip install -t package requests
Move lambda_function.py into the folder (package).
Go to the folder and select all content and zip them.
Go back to the AWS Lambda console. select the function and under the Function code section, click on 'Action' (on the right side) and select Upload a .zip file.
Upload the folder. lambda_function should be uploaded automatically.
Run and Enjoy.
Add a layer to your lambda function
by specifying this arn (ap-south-1)
arn:aws:lambda:ap-south-1:770693421928:layer:Klayers-p38-requests-html:10