I want to upload my a csv file daily on Ambari Apache. I've tried manuplating multiple solutions avaliable online to upload files of Google and other equivalent platforms. I have also tried methods like sftp to help me achieve it, but still have not found a solution. Please recommend any tips, ideas or methods on how should I achieve it.
There is an Ambari method to do this. You can create a custom service in ambari that would run. This would enable you to have ambari self contain the code and execute it. Out of the box Ambari wouldn't technically be running the script, you'd have to run it on a Master/Slave but you might be able to work around that by running an agent on ambari and making it a slave. If it's acceptable you could just have this service installed on one slave and have it push/pull the appropriate file to Ambari.
There are others that have implemented this on just one machine and you can google for how they make sure it's run on just one machine.
Related
I have a windows application built with progress openedge technology.
I have created a python script to generate an excel file but I need to deploy it to the client and im afraid of requiring special permissions on the client side if I compile it to .exe and attempt to run it.
Can someone suggest me a method to be able to integrate python with my project smoothly without breaking anything?
You could compile it on your own machine then try to run it while logged in as a guest user. If a guest account can run it without complaints it will probably run fine on the client machine.
This is crude because you still haven't tested all possible client platforms (unless you're talking about one specific client), also we don't know what's inside your script.
Use icacls to set appropriate permissions of your compiled script before shipping.
I'm not sure about the special permissions thing, but is it possible for you to turn your script into a CGI program and stick it on your webserver, or wrapper it with WebSpeed? Then your app could call a web service to get the .xls file.
I have a folder with different folders in it.
In one of the folder I have a python script.
The python script reads an excel file (which is in the same folder), scrapes information from the internet, updates the excel file and creates another excel file in the main directory.
My question is:
As I can't run my computer non stop, I imagine it's possible (easy? and free) to upload all my folders on a website which will allow me to run my python (3.8) script. Do you have any suggestions ? Which website could be appropriate ? Pythonanywhere.com ?
Plus, I'd like to run this script every morning at 6am.
Thank you for your answers ! :)
Yes, you could use PythonAnywhere -- free accounts allow you to create one scheduled task, which can run once a day. If you have an account, you can set it up on the "Tasks" page.
Some public cloud providers, such as GCP, AWS, and Azure, offer free tier VMs. Simply run the code on those and set up a cron job. Though the network usage probably still costs you a few cents a month, this is a very cheap way to go. You could also consider setting up a FaaS solution against very low cost.
As #Klaus said, this is not a programming question. If you are on linux you can use crontab to schedule your process.
crontab
And if you want to run it on the cloud you can use free services like Heroku
I am looking for a web server, where I can upload files and download files from Ubuntu to Windows and the vice versa. I've builded a web server with Python and I share my folder in Ubuntu and download the files in this folder at Windows. Now I want to look up every millisecond if there is a new file and download this new files automatically. Is there any script or something helpfully for me?
Is a python web server a good solution?
There are many ways to synchronise folders, even remote.
If you need to stick with the python server approach for some reason, look for file system events libraries to trigger your upload code (for example watchdog).
But if not, it may be simpler to use tools like rsync + inotify, or simply lsync.
Good luck!
Edit: I just realized you want linux->windows sync, not the other way around, so since you don't have ssh server on the target (windows), rsync and lsync will not work for you, you probably need smbclient. In python, consider pysmbc or PySmbClient
It is really weird that after clicking run button, it does nothing and also no log and show a clock sign on the first column.
It works normally before. However, after I messed up my python environment, the google coursebuilder can't run web application. That's my guessing. When I run which python.it only shows:
/Library/Frameworks/Python.framework/Versions/2.7/bin/python
These let me feel like I have no way to solve it!Are there anyone who came across this problem before? Any ideas or suggestions?
Updated: I follow suggestions to use command line to run web application on GAE. It reminds me here:
Update: The error message shows that GAE can't get the allocated port and domain. The reason why it happens is that when I use command line to run the web application, I also open GAE GUI to run a web app with the same port number.
So the way to solve it is to close the GAE GUI and free the port. Or we also could designate another kind of port number with command line.(--port=XXXX and --admin_port=YYYY). Or take a look at the doc:
Again thanks for the help of Mihail R!
The OP had multiple issues with GAE setup which were resolved by simply reinstalling the GAE Launcher and making sure the app was first copied into Applications from the .dmg file, then ran from the Applications instead of from inside the .dmg file, and appropriate permissions were suppose to be given so that GAE Launcher created the symlinks it needed to work properly.
More instructions on proper GAE SDK installation can be found here: https://cloud.google.com/appengine/downloads after clicking on the needed SDK and then the OS the SDK will be installed on.
I have a Django application that I would like to deploy to the desktop. I have read a little on this and see that one way is to use freeze. I have used this with varying success in the past for Python applications, but am not convinced it is the best approach for a Django application.
My questions are: what are some successful methods you have used for deploying Django applications? Is there a de facto standard method? Have you hit any dead ends? I need a cross platform solution.
I did this a couple years ago for a Django app running as a local daemon. It was launched by Twisted and wrapped by py2app for Mac and py2exe for Windows. There was both a browser as well as an Air front-end hitting it. It worked pretty well for the most part but I didn't get to deploy it out in the wild because the larger project got postponed. It's been a while and I'm a bit rusty on the details, but here are a few tips:
IIRC, the most problematic thing was Python loading C extensions. I had an Intel assembler module written with C "asm" commands that I needed to load to get low-level system data. That took a while to get working across both platforms. If you can, try to avoid C extensions.
You'll definitely need an installer. Most likely the app will end up running in the background, so you'll need to mark it as a Windows service, Unix daemon, or Mac launchd application.
In your installer you'll want to provide a way to locate a free local TCP port. You may have to write a little stub routine that the installer runs or use the installer's built-in scripting facility to find a port that hasn't been taken and save it to a config file. You then load the config file inside your settings.py and whatever front-end you're going to deploy. That's the shared port. Or you could just pick a random number and hope no other service on the desktop steps on your toes :-)
If your front-end and back-end are separate apps then you'll need to design an API for them to talk to each other. Make sure you provide a flag to return the data in both raw and human-readable form. It really helps in debugging.
If you want Django to be able to send notifications to the user, you'll want to integrate with something like Growl or get Python for Windows extensions so you can bring up toaster pop-up notifications.
You'll probably want to stick with SQLite for database in which case you'll want to make sure you use semaphores to tackle multiple requests vying for the database (or any other shared resource). If your app is accessed via a browser users can have multiple windows open and hit the app at the same time. If using a custom front-end (native, Air, etc...) then you can control how many instances are running at a given time so it won't be as much of an issue.
You'll also want some sort of access to local system logging facilities since the app will be running in the background and make sure you trap all your exceptions and route it into the syslog. A big hassle was debugging Windows service startup issues. It would have been impossible without system logging.
Be careful about hardcoded paths if you want to stay cross-platform. You may have to rely on the installer to write a config file entry with the actual installation path which you'll have to load up at startup.
Test actual deployment especially across a variety of firewalls. Some of the desktop firewalls get pretty aggressive about blocking access to network services that accept incoming requests.
That's all I can think of. Hope it helps.
If you want a good solution, you should give up on making it cross platform. Your code should all be portable, but your deployment - almost by definition - needs to be platform-specific.
I would recommend using py2exe on Windows, py2app on MacOS X, and building deb packages for Ubuntu with a .desktop file in the right place in the package for an entry to show up in the user's menu. Unfortunately for the last option there's no convenient 'py2deb' or 'py2xdg', but it's pretty easy to make the relevant text file by hand.
And of course, I'd recommend bundling in Twisted as your web server for making the application easily self-contained :).