I'll have a linux machine with a virtual machine installed for Microsoft azure soon. I need to run some data mining/graph analysis algorithms on the azure because I work with big data. I don't want to use azure machine learning stuff. just want to run my own python code. What are the steps? If needed, hoe can I install python libraries on azure?
There is no additional steps to do in comparison to Your own. local server. Linux on Azure is a standard Linux machine. If You are looking for step-by-step hopw to on running Linux VM on Azure, just search on azure.com and You will find it. I think You will not have any problems even without documentation. Azure portal is very simple to use, also CLI tool for Linux, Mac and Windows. You just need to run Linux VM and SSH-in to it. Nothing more. If You need some help, just write here.
Related
I have python scripts for automated trading for currency and I want to deploy them by running on Jupter Lab on a cloud instance. I have no experience with cloud computing or linux, so I have been trying weeks to get into this cloud computing mania, but I found it very difficult to participate in it.
My goal is to set up a full-fledged Python infrastructure on a cloud instance from whichever provider so that I can run my trading bot on the cloud.
I want to set up a cloud instance on whichever provider that has the latest python
installation plus the typically needed scientific packages (such as NumPy and pandas and others) in combination with a password-protected and Secure Sockets Layer (SSL)-encrypted Jupyter
Lab server installation.
So far I have gotten no where. I am currently looking at the digital ocean website for setting jupter lab up but there are so many confusing terms.
What is Ubuntu or Debian? Is it like a sub-variant of Linux operating system? Why do I have only 2 options here? I use neither of the operating system, I use the windows operating system on my laptop and it is also where I developed my python script. Do I need a window server or something?
How can I do this? I tried a lot of tutorials but I just got more confused.
Your question raises several more about what you are trying to accomplish. Are you just trying to run your script on cloud services? Or do you want to schedule a server to spin up and execute your code? Are you running a bot that trades for you? These are just some initial questions after reading your post.
Regarding your specific question regarding Ubuntu and Debian, they are indeed Linux distributions which are popular option for servers. You can set up a Windows server on AWS or another cloud provider, but Linux distributions being much more popular are going to have lots of documentation, articles, stackoverflow posts around a Linux based server.
If you just want to run a script on a cloud on demand, you would probably have a lot of success following Wayne's comment around PythonAnywhere or Google Colab.
If you want your own cloud server, I would suggest starting small and slow with a small or free tier EC2 instance by following a tutorial such as this https://dataschool.com/data-modeling-101/running-jupyter-notebook-on-an-ec2-server/ Alternatively, you could splurge for an AWS AMI which will have much more compute power and be configured.
I have similar problem and the most suiteble solution to me is using docker container for jupyter notebooks. The instructions on how to install Docker can be found at https://docs.docker.com/engine/install/ubuntu/ There is ready to use Docker image docker pull jupyter/datascience-notebook for jupyter notebook python stack. The docker compose files und sone addional insruction you will fid at https://github.com/stefanproell/jupyter-notebook-docker-compose/blob/master/README.md.
Summary: I'm trying to create a python program to run Powershell scripts in Azure Windows VMs. But I can't find good documentation about which libraries of the SDK I can use.
Detailed: In order to automate certain administrative tasks, a python program should run, authenticate into Azure and then run Powershell scripts in certain Windows VMs. I think I have the auth part thanks to azure.identity, but I can't find any library to interact with a running VM. Sure, lots for creating or modifying a VM in Azure, but nothing to interact with it. Neither in the SDK or in the API browser. I know it can be done with Azure CLI, but id like to use the SDK, if possible at all.
My understanding is that you would like to run PowerShell remotely using the Python SDK.
AFAIK, I don't think there is a provision within the Azure Python SDK to run remote PS scripts.
Workaround :
(But please note that this outside the Azure Python SDK.)
You could create Azure Windows WinRM VM template and you could execute ps commands from the Python code using the winrm library like discussed in this thread.
I have a requirement where I have to use the Python libraries I created on my machine, in the cloud, such that whenever any new dataset is loaded, this Python library have to start acting on it.
How can I do this? Where will I put the dataset and the python codes in Azure?
Thanks,
Shyam
There are more possibility to do that.
Run your Python code on Azure Web Apps for Containers—a Linux-based, managed application platform
Azure Functions allows running Python code in a serverless environment that scales on-demand.
Use a managed Hadoop and Spark cluster with Azure HDInsights, suitable for enterprise-grade production workloads.
Use a friction-free data science environment that contains popular tools for data exploration, modeling, and development activities.
Azure Kubernetes Service (AKS) offers a fully-managed Kubernetes cluster to run your Python apps and services, as well as any other Docker container. Easily integrate with other Azure services using Open Service Broker for Azure.
Use your favorite Linux distribution, such as Ubuntu, CentOS, and Debian, or Windows Server. Run your code with scalable Azure Virtual Machines and Virtual Machine Scale Sets.
Run your own Python data science experiments using a fully-managed Jupyter notebook with Azure Notebooks.
The easiest and fastest way to run your code is 1. option. Create a web app and a web job in there.
I'm new to Spark.I installed a Spark 2.3.0 in Stand-Alone-Mode on an Ubuntu 16.04.3 server. That runs well so far. Now I would like to start developing with pyspark because I've got more experience using python than scala.
Ok. Even after using google for a while I'm not sure how I should setup my development environment. My local machine is a windows 10 laptop with eclipse neon and pydev configured. What are the neccessary steps to set ist up that I can develop in a local context and submit my modules to the spark cluster on my server?
Thank for helping.
use spark-submit to run locally or on a cluster. There are many online tutorials for this. I like the AWS documentation which explains the architecture, has sample spark code, and gives examples of local and remote commands. Even if you are not using AWS EMR the basics are the same.
give it a try and let us know how it goes
I have developed a web-based application for end-users that resides in a docker container. The container itself hosts a few python dependencies, a few public repositories, and a flask based web front-end with a MongoDB back-end that is started when the container is initialized.
It's fairly straightforward to download the container and run it on a docker host. However, most docker hosts (if not all) are not free.
Therefore, if an end-user wanted to use my application off the cloud they would have to download and install docker and associated dependencies on their local machine prior to being able to use the image (which is even more complicated on a system like Windows or Mac OSX)
With that being said, my question is: Is there any tool that has been developed to help ease this requirement on the end-user for deployment to users local desktop environments? I understand installing and using docker is not THAT hard, but some people are still very afraid of command-lines and I was hoping to find a method that would help alleviate some of these 'scary' requirements.
Did you look at Boot2Docker? It packages up the Docker CLI compiled for Windows or OSX, a VirtualBox VM to run Linux for the containers, with an easy-to-use installer.
Also https://kitematic.com adds more point-and-click for Mac users.
Overall, however, Docker is a developer/devops tool, and I haven't seen much aimed at helping non-technical folks use it.