I have python scripts for automated trading for currency and I want to deploy them by running on Jupter Lab on a cloud instance. I have no experience with cloud computing or linux, so I have been trying weeks to get into this cloud computing mania, but I found it very difficult to participate in it.
My goal is to set up a full-fledged Python infrastructure on a cloud instance from whichever provider so that I can run my trading bot on the cloud.
I want to set up a cloud instance on whichever provider that has the latest python
installation plus the typically needed scientific packages (such as NumPy and pandas and others) in combination with a password-protected and Secure Sockets Layer (SSL)-encrypted Jupyter
Lab server installation.
So far I have gotten no where. I am currently looking at the digital ocean website for setting jupter lab up but there are so many confusing terms.
What is Ubuntu or Debian? Is it like a sub-variant of Linux operating system? Why do I have only 2 options here? I use neither of the operating system, I use the windows operating system on my laptop and it is also where I developed my python script. Do I need a window server or something?
How can I do this? I tried a lot of tutorials but I just got more confused.
Your question raises several more about what you are trying to accomplish. Are you just trying to run your script on cloud services? Or do you want to schedule a server to spin up and execute your code? Are you running a bot that trades for you? These are just some initial questions after reading your post.
Regarding your specific question regarding Ubuntu and Debian, they are indeed Linux distributions which are popular option for servers. You can set up a Windows server on AWS or another cloud provider, but Linux distributions being much more popular are going to have lots of documentation, articles, stackoverflow posts around a Linux based server.
If you just want to run a script on a cloud on demand, you would probably have a lot of success following Wayne's comment around PythonAnywhere or Google Colab.
If you want your own cloud server, I would suggest starting small and slow with a small or free tier EC2 instance by following a tutorial such as this https://dataschool.com/data-modeling-101/running-jupyter-notebook-on-an-ec2-server/ Alternatively, you could splurge for an AWS AMI which will have much more compute power and be configured.
I have similar problem and the most suiteble solution to me is using docker container for jupyter notebooks. The instructions on how to install Docker can be found at https://docs.docker.com/engine/install/ubuntu/ There is ready to use Docker image docker pull jupyter/datascience-notebook for jupyter notebook python stack. The docker compose files und sone addional insruction you will fid at https://github.com/stefanproell/jupyter-notebook-docker-compose/blob/master/README.md.
Related
I have a fx vol model which I have written in python / questdb on my local machine. The standalone model works as an application in my machine and I also have lots of feeds written which are constantly updating the local machine questdb instance.
Now I want to move to web and have my application and database on the web server away from my machine.
I am not too familiar of web servers and how to install a questdb there.
From my knowledge I will need :
a VPS paid subscription where I have centOS and I have python support.
I need to install questdb here ( using docker ?)
Install all my python model here
Start the questdb in the VPS
Configure my scripts to use the hosted questdb
The model saves output in a table in questdb ; while the scripts also keep on updating new feeds to questdb
Start a webserver which provides a web access to the model results saved in the questdb
7.a I need to provide username and login for the website
7.b I need to use some vitualization
7.c I need the users once they are logged in to run some simulations
What I need some quidance is :
What sort of VPS service to look for
Has any one already installed a questdb in this way
Which webserver is the best for python
Lots of good questions there. Will try to help with all of them (disclaimer, I work for QuestDB as a developer advocate)
What sort of VPS service to look for:
Anything will work, but depending on the volume you are going to ingest/query, you will greatly benefit from a fast drive. In many cases the default disk drive for a VPS/Cloud provider will be slower than the SSD drive on your local development machine. See if the default works for you and otherwise select a better option when starting your virtual machine.
Has any one already installed a questdb in this way
Sure. If you want to install on AWS, Google Cloud, or Digital Ocean, you have detailed instructions at the QuestDB site. If you prefer to use Docker, it is fully supported, but a popular option is installing the binaries and just start QuestDB as a service.
Which webserver is the best for python
This one is tricky, as it really depends on what you will do with it. From your question I am getting the idea you want to run a small user-facing web application that will also run some background jobs. For that scenario a suitable framework can be Flask, which also offers some plugins for user/login management and for background tasks (using redis as a dependency)
I have a requirement where I have to use the Python libraries I created on my machine, in the cloud, such that whenever any new dataset is loaded, this Python library have to start acting on it.
How can I do this? Where will I put the dataset and the python codes in Azure?
Thanks,
Shyam
There are more possibility to do that.
Run your Python code on Azure Web Apps for Containers—a Linux-based, managed application platform
Azure Functions allows running Python code in a serverless environment that scales on-demand.
Use a managed Hadoop and Spark cluster with Azure HDInsights, suitable for enterprise-grade production workloads.
Use a friction-free data science environment that contains popular tools for data exploration, modeling, and development activities.
Azure Kubernetes Service (AKS) offers a fully-managed Kubernetes cluster to run your Python apps and services, as well as any other Docker container. Easily integrate with other Azure services using Open Service Broker for Azure.
Use your favorite Linux distribution, such as Ubuntu, CentOS, and Debian, or Windows Server. Run your code with scalable Azure Virtual Machines and Virtual Machine Scale Sets.
Run your own Python data science experiments using a fully-managed Jupyter notebook with Azure Notebooks.
The easiest and fastest way to run your code is 1. option. Create a web app and a web job in there.
I'm new to Spark.I installed a Spark 2.3.0 in Stand-Alone-Mode on an Ubuntu 16.04.3 server. That runs well so far. Now I would like to start developing with pyspark because I've got more experience using python than scala.
Ok. Even after using google for a while I'm not sure how I should setup my development environment. My local machine is a windows 10 laptop with eclipse neon and pydev configured. What are the neccessary steps to set ist up that I can develop in a local context and submit my modules to the spark cluster on my server?
Thank for helping.
use spark-submit to run locally or on a cluster. There are many online tutorials for this. I like the AWS documentation which explains the architecture, has sample spark code, and gives examples of local and remote commands. Even if you are not using AWS EMR the basics are the same.
give it a try and let us know how it goes
I have an apps coded in python/pandas/scipy which can be launched by anyone authorized. I want to use Google Cloud Platform to host it but I can't find a good way to set up this.
Since I want my app to be a web app, part of this is hosted on google app engine, but since google app engine does not seem compatible with big calculus and scientific libraries, I think I shall send the task to a VM with the scientific libraries.
My questions are:
1- could I create a VM each time the app is launched, in order to save money? But if I do this, I have to setup pandas scipy etc... each time, which should take some time?
2- Am I condemned to have a VM running every time and activated by the app? But if two people launch the calculus together, this can be really bad perf?
3 - Shall I package my app as a .exe and launch it as a standalone?
I am totally lost on how to handle such an architecture, can anyone give me some advices?
Thank you!
You can host your application on Managed VMs. Applications that run on managed VMs are not subject to the restrictions imposed on the sandboxed runtimes (Java, Python, PHP, and Go).
You can also choose the hosting environment (sandboxed or managed VM) separately for each module in your application. This means an application can contain a mix of modules that use different hosting environments. For instance, you might use the sandbox environment for your client-facing frontend, and use a module running in a managed VM for backend processing.
I'll have a linux machine with a virtual machine installed for Microsoft azure soon. I need to run some data mining/graph analysis algorithms on the azure because I work with big data. I don't want to use azure machine learning stuff. just want to run my own python code. What are the steps? If needed, hoe can I install python libraries on azure?
There is no additional steps to do in comparison to Your own. local server. Linux on Azure is a standard Linux machine. If You are looking for step-by-step hopw to on running Linux VM on Azure, just search on azure.com and You will find it. I think You will not have any problems even without documentation. Azure portal is very simple to use, also CLI tool for Linux, Mac and Windows. You just need to run Linux VM and SSH-in to it. Nothing more. If You need some help, just write here.