I deployed a project 3 days ago with shub deploy which ran perfectly. I just tried deploying the same code again today and it shows a requirements error like this:
Packing version c1f72fb-master
Deploying to Scrapy Cloud project "187201"
Deploy log last 30 lines:
---> 72b41733c189
Step 9 : RUN mkdir /app/python && chown nobody:nogroup /app/python
---> Using cache
---> dda1555878eb
Step 10 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install
--user --no-cache-dir -r /app/requirements.txt
---> Using cache
---> cccdde466280
Step 11 : COPY *.egg /app/
---> afc6b3540c92
Removing intermediate container bd3bedcee848
Step 12 : RUN if [ -d "/app/addons_eggs" ]; then rm -f /app/*.dash-
addon.egg; fi
---> Running in 80461e4402dc
---> 830db9615167
Removing intermediate container 80461e4402dc
Step 13 : ENV PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
---> Running in 9af6ab0fdc02
---> 0f33ba992cc7
Removing intermediate container 9af6ab0fdc02
Successfully built 0f33ba992cc7
>>> Checking python dependencies
WARNING: There're some errors when doing pip-check:
Traceback (most recent call last):
File "/usr/local/bin/pip", line 4, in <module>
import re
File "/usr/local/lib/python3.6/re.py", line 142, in <module>
class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
{"message": "Dependencies check exit code: 1", "details": "Pip checks
failed, please fix the conflicts", "error": "requirements_error"}
{"status": "error", "message": "Requirements error"}
Deploy log location:
c:\users\sim04\appdata\local\temp\shub_deploy__oqwt2.log
Error: Deploy failed: {"status": "error", "message": "Requirements error"}
Make sure you followed these steps to specify your required modules correctly.
Create a file named scrapinghub.yml in your project's main folder with following contents.
projects:
default: 111149
requirements:
file: requirements.txt
Where 111149 is my proejct ID on scrapinghub.
Create another file named requirements.txt in same directory.
and put your required modules along with the version number you are using in that file like so,
MySQL-python==1.2.5
PS: I was using MySQLDB module so I put that.
Related
When trying to synthesize my CDK app, I receive the following error:
`
Traceback (most recent call last):
File "C:\Users\myusername\PycharmProjects\rbds-cdk_testing\app.py", line 2, in <module>
from aws_cdk.core import App, Environment
File "C:\Users\myusername\PycharmProjects\rbds-cdk_testing\.venv\lib\site-packages\aws_cdk\__init__.py", line 1260, in <module>
from .cloud_assembly_schema import (
ImportError: cannot import name 'AssetManifestOptions' from 'aws_cdk.cloud_assembly_schema' (C:\Users\myusername\PycharmProjects\rbds-cdk_testing\.venv\lib\site-packages\aws_cdk\cloud_assembly_schema\__init__.py)
I am using node version 18.0.0. Here's the steps I've done in creating my CDK app:
(FROM c:\Users\myusername\)
installed nvm
installed npm
nvm use 18.0.0
npm install -g yarn
npm install -g aws-cdk
cdk bootstrap aws://account-number/region
cd .\PyCharmProjects\mycdkapp
cdk init app --language python
.venv\Scripts\activate.bat
python -m pip install aws-cdk.aws-glue
python -m pip install aws-cdk
I error out even when executing cdk ls as the runtime tries to run app.py which contains
\
import yaml
from aws_cdk.core import App, Environment
from pipeline import PipelineCDKStack
In checking whether the init.py file for aws_cdk contains AssetManifestOptions, I've discovered it is completely missing:
Am I missing something here or is this a unique bug that I am experiencing? Any help much appreciated! I am banging my head on this one.
Its the same here, I think the issue can be in wrong package version.
cloud-assembly-schema==2.50.0 contains AssetManifestOptions.
Can you please paste here output of
pip list -v | grep aws
Iam able to install 2.50.0, however it depends on other packages of the same version (see attach)
And I cant set up core package because there is no CDKv2 matching distribution at the moment
I am trying to synthesize a CDK app (typeScript) which has some python lambda functions.
I am using PythonFunction to use a requirements.txt file to install the external dependencies. I am running vscode on WSL. I am encountering the following error.
Bundling asset Test/test-lambda-stack/test-subscriber-data-validator-poc/Code/Stage...
node:internal/fs/utils:347
throw err;
^
Error: ENOENT: no such file or directory, open '~/.nvm/versions/node/v16.17.0/lib/node_modules/docker/node_modules/highlight.js/styles/cp -rTL /asset-input/ /asset-output && cd /asset-output && python -m pip install -r requirements.txt -t /asset-output.css'
at Object.openSync (node:fs:594:3)
at Object.readFileSync (node:fs:462:35)
at module.exports (~/.nvm/versions/node/v16.17.0/lib/node_modules/docker/src/getColourScheme.js:47:26)
at ~/.nvm/versions/node/v16.17.0/lib/node_modules/docker/src/docker.js:809:47
at FSReqCallback.readFileAfterClose [as oncomplete] (node:internal/fs/read_file_context:68:3)
at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17) {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: '~/.nvm/versions/node/v16.17.0/lib/node_modules/docker/node_modules/highlight.js/styles/cp -rTL /asset-input/ /asset-output && cd /asset-output && python -m pip install -r requirements.txt -t /asset-output.css'
}
Error: Failed to bundle asset Test/test-lambda-stack/test-subscriber-data-validator-poc/Code/Stage, bundle output is located at ~/Code/AWS/CDK/test-dev-poc/cdk.out/asset.6b577fe604573a3b53e635f09f768df3f87ad6651b18e9f628c2a086a525bb49-error: Error: docker exited with status 1
at AssetStaging.bundle (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:614)
at AssetStaging.stageByBundling (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:4506)
at stageThisAsset (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:1867)
at Cache.obtain (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/core/lib/private/cache.js:1:242)
at new AssetStaging (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:2262)
at new Asset (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.js:1:736)
at AssetCode.bind (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/aws-lambda/lib/code.js:1:4628)
at new Function (~/Code/AWS/CDK/test-dev-poc/node_modules/aws-cdk-lib/aws-lambda/lib/function.js:1:2803)
at new PythonFunction (~/Code/AWS/CDK/test-dev-poc/node_modules/#aws-cdk/aws-lambda-python-alpha/lib/function.ts:73:5)
at new lambdaInfraStack (~/Code/AWS/CDK/test-dev-poc/lib/serviceInfraStacks/lambda-infra-stack.ts:24:40)
My requirements.txt file looks like this
attrs==22.1.0
jsonschema==4.16.0
pyrsistent==0.18.1
My cdk code is this
new PythonFunction(this,`${appName}-subscriber-data-validator-${stage}`,{
runtime: Runtime.PYTHON_3_9,
entry: join('lambdas/subscriber_data_validator'),
handler: 'lambda_hander',
index: 'subscriber_data_validator.py'
})
Do I need to install anything additional? I have esbuild installed as a devDependency. Having a real hard time getting this work. Any help is appreciated.
I am trying to deploy a basic web application using Elastic Beanstalk from AWS.
My app is written in python and uses pyTorch library so it can import NLP model named "bart-cnn-large" (with it I can produce text summarization).
I have a file named requirements.txt and with it the EC2 sets the virtual environment.
However, it always fails when trying to install the pytorch library.
If I remove "torch" from the requirements.txt then the installation doesn't fail anymore.
but I get this message:
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models
won't be available and only tokenizers, configuration and file/data
utilities can be used.
If I leave "torch" in requirements I get this message:
2021/08/06 10:54:23.688955 [ERROR] An error occurred during execution
of command [app-deploy] - [InstallDependency]. Stop running the
command. Error: fail to install dependencies with requirements.txt
file with error Command /bin/sh -c
/var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt
failed with error exit status 1. Stderr:ERROR: Invalid requirement:
'torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0'
(from line 1 of requirements.txt)
the requirments.txt content:
>torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f
https://download.pytorch.org/whl/torch_stable.html
Flask~=2.0.1
Werkzeug~=2.0.1
tika~=1.24
beautifulsoup4~=4.8.2
docx2txt~=0.8
transformers~=4.8.2
clean-text
I tried several versions of "pip torch" but none seems to work.
is this a storage problem? why won't it install?
Im trying to run a package wal-e which I have installed as user root with sudo python3 -m pip install wal-e[aws,azure,google,swift].
I can run this command perfectly as user root using envdir /etc/wal-e.d/env wal-e backup-fetch /var/lib/postgresql/9.6/main LATEST.
However, when I sudo su - postgres and then run envdir /etc/wal-e.d/env wal-e backup-fetch /var/lib/postgresql/9.6/main LATEST, I get the error
Traceback (most recent call last):
File "/usr/local/bin/wal-e", line 7, in <module>
from wal_e.cmd import main
ImportError: No module named 'wal_e.cmd'
I gave user postgres full sudo permissions with usermod -aG sudo postgres. Also the wal-e package is installed in the same location.
When I run ls -la I get
-rwxr-xr-x 1 root root 211 Sep 20 14:24 /usr/local/bin/wal-e
Im also on Ubuntu 16.04.3
How can I run the command just like the root user?
I had to run a strict setup process for wal-e in order for the package to function properly.
Virtually what it boiled down to was installing all necessary dependencies on the machine that I was working with before installing and creating the user postgres. If the user was created before all the dependencies were installed, I got permissions errors.
I'm trying to deploy my spider to Scrapy Cloud using shub but I keep running into this following error:
$ shub deploy
Packing version 2df64a0-master
Deploying to Scrapy Cloud project "164526"
Deploy log last 30 lines:
---> Using cache
---> 55d64858a2f3
Step 11 : RUN mkdir /app/python && chown nobody:nogroup /app/python
---> Using cache
---> 2ae4ff90489a
Step 12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install --user --no-cache-dir -r /app/requirements.txt
---> Using cache
---> 51f233d54a01
Step 13 : COPY *.egg /app/
---> e2aa1fc31f89
Removing intermediate container 5f0a6cb53597
Step 14 : RUN if [ -d "/app/addons_eggs" ]; then rm -f /app/*.dash-addon.egg; fi
---> Running in 3a2b2bbc1a73
---> af8905101e32
Removing intermediate container 3a2b2bbc1a73
Step 15 : ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
---> Running in ccffea3009a4
---> b4882513b76e
Removing intermediate container ccffea3009a4
Successfully built b4882513b76e
>>> Checking python dependencies
scrapinghub 1.9.0 has requirement six>=1.10.0, but you have six 1.7.3.
monkeylearn 0.3.5 has requirement requests>=2.8.1, but you have requests 2.3.0.
monkeylearn 0.3.5 has requirement six>=1.10.0, but you have six 1.7.3.
hubstorage 0.23.6 has requirement six>=1.10.0, but you have six 1.7.3.
Warning: Pip checks failed, please fix the conflicts.
Process terminated with exit code 1, signal None, status=0x0100
{"message": "Dependencies check exit code: 193", "details": "Pip checks failed, please fix the conflicts", "error": "requirements_error"}
{"message": "Requirements error", "status": "error"}
Deploy log location: /var/folders/w0/5w7rddxn28l2ywk5m6jwp7380000gn/T/shub_deploy_xi_w3xx8.log
Error: Deploy failed: b'{"message": "Requirements error", "status": "error"}'
It looks like a simple problem of an outdated package (six). However the installed package actually IS up to date:
$ pip show six
Name: six
Version: 1.10.0
Summary: Python 2 and 3 compatibility utilities
Home-page: http://pypi.python.org/pypi/six/
Author: Benjamin Peterson
Author-email: benjamin#python.org
License: MIT
Location: /Users/mac/.pyenv/versions/3.6.0/lib/python3.6/site-packages
Requires:
I'm running python 3.6 through pyenv on a Mac.
Any ideas?
EDIT:
my requirements.txt file only contains the following dependency:
newspaper==0.0.9.8
EDIT 2: scrapinghub.yml
projects:
default: 164526
requirements_file: requirements.txt
Thanks,
Simon!
Managed to solve this (with help from scrapinghub's support forum) by adding the following code to scrapinghub.yml:
stacks:
default: scrapy:1.3-py3
and changing requirements.txt to use the python3 branch of newspaper:
newspaper3k==0.1.9