See TL;DR before the question if you wish, but any quick responses are appreciated.
I have a web-app that uses Node.js for the backend and Python for running a particular script.
The app basically takes an MS Excel file, does some computations on it (here is where Python comes into play) and returns the result MS Excel file.
I've used Python successfully along with Node.js using the child_process class in Node.js.
I've also deployed the app on Heroku using Heroku buildpacks . (I've added both Python and Node.js buildpacks in order for it to work.)
Now, the slug size limit of Heroku is a problem for me as my Python program (which is basically an ML model) uses resources more than 300mb (around 13.8k individual files). Before someone points it out, I cannot use S3 in this case as my Python script needs to read and use the files (which are the parameters for the ML model) from a folder.
So, I thought of moving to AWS Elastic Beanstalk but I haven't been able to find out any features analogous to Heroku buildpacks. Is there any way I can make it work?
TL;DR
I want to deploy an app that uses Node.js along with Python with some (heavy) dependencies.
Can I do that? If yes then how? Are there any features like Heroku buildpacks?
Any detailed pointers are welcome.
Thank you =)
Related
i am new in developing web apps so i might get confused a lot of times!
The problem is this:
I was developing with Pycharm son sort of basic social network, and at first when you sign up the users were created in local folder as JSON files and then i look foward to make a deployment, and i did it without problem using PythonAnywere (PA). Lets call my .py file "server.py" where i have the whole thing.
Then i started looking for some cloud service and I ended modifying everything in order to work with Mongodb Atlas and it was a complete success. I made a lot of local test using Pycharm and everything is OK, the users are now created on cloud service.
My problem is that i would like to make a deployment test with that Mongodb service version, and i was trying to use (PA) again but this time its give me a lot of errors.
Note: i already install all the requirements in (PA) from pip freeze requirements.txt
Is there a problem with PA and MongoDB? is ther any other better option?
Should it run ok if the first version of "server.py" was ok?
I just replaced that file with new one, that was runnning perfect on localhost.
If you need more info just tell me, i am very new in this.
Thanks a lot
Just started using Google Cloud SDK Shell after using the older, gui-based, version. I have multiple projects under development, if that matters.
Here's what I do
run gcloud SDK shell (click on the icon!)
cd \myproject
dev_appserver.py app.yaml
In the browser (Chrome),
browse to http://localhost:8000/datastore
Under Datastore Viewer, I see 'tables' from a completely different project
(say, myotherproject)
Under Datastore Indexes, I see 'indexes' from the correct project (myproject)
Under Task Queues, I see the correct queues listed (I have specified different queues setup for parts of myproject)
Everything works fine for myotherproject. So, is there something I am missing to get the Datastore Viewer to show the correct 'tables'?
Many thanks, David
Edit: no matter what project I run, Datastore Viewer shows the same data (from myotherproject) but Datastore Indexes show the correct indexes.
Edit: Windows 8.1, Python v2.7.13:a06454b1afa1
Edit: further questions 1) does gcloud sdk use a different datastore from the original app engine sdk? 2) if so, where is it by default or do I have to define it upfront?
Thanks to everyone for their help with this. It appears GCloud uses one datastore for all projects so the --datastore_path is not really optional when you have multiple paths. However, I kept getting errors with --datastore_path so I went with the following...
dev_appserver.py --storage_path=c:\gcdata\projectname app.yaml
Yes, could have been c:\temp but this gives me separate 'databases', one for each project.
Note also that GCloud SDK does not use the same data as the original App Engine SDK grrrrrr!
Say I have a file "main.py" and I just want it to run at 10 minute intervals, but not on my computer. The only external libraries the file uses are mysql.connector and pip requests.
Things I've tried:
PythonAnywhere - free tier is too limiting (need to connect to external DB)
AWS Lambda - Only supports up to Python 2.7, converted my code but still had issues
Google Cloud Platform + Heroku - can only find tutorials covering deploying applications, I think these could do what I'm looking for but I can't figure out how.
Thanks!
I'd start by taking a look at this question/answer that I asked previously on unix.stackexchange - I went with an AWS redhat installation and it was free to use.
Once you've decided on your VM, you can add SSH onto your server using any SSH client and upload your Python script. A personal preference is this application.
If you need to update the Python version on the server, you can do this by installing the required Python RPMs. A quick google should return the yum [or whichever RPM management system you're using] repository for the required RPMs.
Once you've installed the version of Python that you need, I'd suggest looking into the 'crontab' which can be used to schedule jobs. You can set a cronjob to run every 10minutes which will call your script.
See this site for more information on how to use the crontab
This sounds like a perfect use case for AWS Lambda which supports Python. You can invoke your Lambda on a schedule using Scheduled Events.
I see that you tried Lambda and it didn't work out for you which is too bad as that seems like the easiest route. You could also launch an EC2 instance and use userdata to schedule a cron when the instance starts.
Another option would be an Elastic Beanstalk worker with a cron.yml that defines your schedule. Elastic Beanstalk supports Python 3.4.
Update: AWS does now support Python 3.6. Just select Python 3.6 from the runtime environments when configuring.
I have been given a task to complete: Deploy my pre-existing Pyramid application onto our EC2 Linux server. I would like to do this with a minimal amount of stress and error, especially considering am I totally new to AWS.
What I have done so far:
Setup the EC2 instance which I can SSH into.
Locally develop my Pyramid application
And, we version control the application with GitHub.
We are using: Pyramid (latest), along with Python 2.7.5 and Postgresql (via SQLAlchemy and Alembic.)
What is a basic, high-level list of steps to ensure that my application is deployed appropriately?
Where, if at all, does something like Elastic Beanstalk come into play?
And, considering my project is currently in a Git repo, what steps or considerations must be taken to accommodate this?
I'm not looking for opinions on how to tweak my setup or anything like that. I am looking for a non-debatable, comprehensible set of steps or considerations to deploy my application in the most basic form. This server is for development purposes only, so I am not looking for a full-blown solution.
I have researched this topic for Django projects, and frankly, I am a bit overwhelmed with the amount of different possible options. I am trying to boil this situation down to its minimal components.
I appreciate the time and help.
Deploying to an EC2 server is just like deploying to any other Linux server.
If you want to put it behind a load balancer, you can do which is fully documented.
You can also deploy to Elastic Beanstalk. Where as EC2 is a normal Linux sever, Beanstalk is more like deploying to an environment, you just push all your git changes into an S3 repo, your app then gets built and deployed onto beanstalk.
Meaning no server setups, no configuration (other than the very basics) and all new changes you push to S3, get built and update each version of your app that may have been launched on beanstalk.
You don't want to host your database server on EC2, use Amazons RDS database server, dead simple and takes about two minutes to setup and configure.
As far as file storage goes, move everything to S3.
EC2 and beanstalk should not be used for any form of storage.
I would suggest to run two instances and use Elastic Load Balancer.
Never run anything important on a single EC2 instance, EC2 instances are not durable, they can suddenly vanish, taking whatever data you had stored on it.
Everything else should work as in Pyramid Cookbook description.
Although there is no specific recipe for deploying a Pyramid application to AWS in the Pyramid Cookbook, there are other recipes that may offer a partial solution.
I'm working on a Heroku app built in Python, and I can't find a recommended way to add a step to deployment for concatenating/processing/minifying JavaScript and CSS assets. For example, I might like to use tools like r.js or less.
I've seen something called "collectstatic" that Heroku knows to run for Django apps, but my application is using web.py, not Django.
One less-than-perfect approach would be to use these tools locally, on my development machine, to produce combined/compressed static assets. I could then check those compiled files into the git repository and push them to Heroku.
Is there any support for this kind of step built in to Heroku? What is the best way of handling javascript/css files for Heroku web apps in Python?
Using the buildpack-multi, Heroku lets you run multiple buildpacks. You can either create our own buildpack that only does the asset compilation that you need or find one that already does it and layer it on top of the Python buildpack with buildpack-multi.
I would generally recommend your less than perfect approach, especially if you have a small number of files.
Simplicity is always better than functionality.