I need to setup a Heroku app (python) which would perform scheduled tasks that include fetching a set of data (.csv an .html) files from other Heroku app (ROR) and returning a result back to that app.
Also it should be restricted only to my app to be able to connect to the ROR app because it deals with sensitive information. There would be from 20 to 100 files each time so I want them to be compressed somehow to transfer them quiclky (to avoid bothering the server for too long).
I'm interested in possible ways to accomplish it. The first thought is to send HTTP GET request to the ROR app and fetch the necessary files yet it generally not secured at all. Would SCP work in some way in this situation or you have any other ideas?
Thanks in advance!
I would suggest writing a secured JSON or XML API to transfer the data from app to app,. Once the data is received I would then generate the .csv or .html files from the received data. It keeps things clean and easy to modify for future revisions because now you'll have an API to interact with.
Related
Problem: Habitica is a habit-tracking app, but its personal data logs are not as detailed as I want. I want to create a local log of when I mark off habits/todo's in the app. Habitica offers certain webhooks that trigger when habits/todo's are checked off, which seems perfect for what I want, but how do I turn these triggers into a local log? I would like to use Python for this.
Ideas: It seems to me that I would need to set up some kind of personal cloud server to receive this data, turn it into a log, and then store it for download. I have previously deployed a Flask app using Heroku, so if this could be done similarly, that would be ideal. However, I don't know much about this, so I would welcome any ideas or advice.
Creating the Habitica webhook as Flask application is a good approach.
Heroku supports Python/Flask very nicely however the file system is ephemeral, hence it gets wiped out at every application restart.
In order to persist data you can look at various options:
save the file to AWS S3
save the data into a DB (Heroku has a free plan for PostgreSQL)
I have a running python application that needs to receive some data and process them. and I also have a PHP server that can get these data. I want to send JSON data from PHP to my python app.
anyway except running a python web server and send data to it, or insert into DB and get from DB with python?
thanks.
I tried using python cherryPy web server.
#Niklas D It would be easier to answer your question, if you can give some more context about the application or use case you want to solve.
Some further possibilities are:
Glue Code (I never did it with python and php only C++ with python, but you should be able to find examples on the internet e.g. https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages#PHP )
Messaging Systems like RabbitMQ, ActiveMQ, ZeroMQ, etc.
Redis (I know you said except writing to a database, but Redis provides some features for publish subscribe https://redis.io/commands/pubsub which allows you to write to Redis from the one side and get data on the other side without polling the db all the time, which is the issue you have with using a database I guess) It's a bit easier to setup and use, than a messaging system.
TCP connection between the python and php application. https://medium.com/swlh/lets-write-a-chat-app-in-python-f6783a9ac170
If you want to send data to a python application using web protocols, i.e send POST, GET requests etc then you need to create a python web app to receive and handle those requests. Which in turn needs to be running off a webserver or you could build serverless functions to handle this, see https://serverless.com/
If you want to get data using a python application, i.e the python app sends POST and GET requests etc to your php app to ask for the JSON payload you can build an app using python's standard requests library https://docs.python.org/3/library/urllib.request.html or better still us the Requests package http://docs.python-requests.org/en/master/
Or you could do something and save the JSON file to disk and then open it with your python app. You'd need to set up scheduling or make your php app execute python code on the server... This last suggestion is a bad idea please don't unless your app is isolated and not publicly accessible or you know how to lock down your security.
I used to do this a while back (about 3 years ago). I don't remember how, so maybe someone here can help me.
I create some entities in localhost datastore. Now I need to transfer the entities to the production datastore. I have existing entities in the store that I do not wish to delete. I just want to admit the additional data to production. Any ideas how I might do this?
You can enable remote API for your application, then use a locally running program to auth to the remote API and do datastore operations. This is outlined on the docs paged linked.
I have a server which runs flask with python.
Now I want to make an application which can do various tasks like uploading files, updating redis database and various other things.
Now ofcourse this could be done using html pages but since the operation could involve lots of files realtime input of data and other things it might be better to make an application and manage the server from that point rather than webpages.
do you suggest using webpages anyway or would you make an application for it?
and if I make an application should I use http or not?
sorry if this is a uninformed question but I would like to learn the best methods
You might want to look into Flask-Script. It allows you to run various commands related to your flask application easily. It also allows you to easily add your own commands to it. This way you will be able to keep your administrative code still within the Flask app, but not necessarily have it accessible via a web page.
I want to upload an image to the blobstore, because I want to support files larger than 1MB. Now, the only way I can find is for the client to issue a POST where it sends the metadata, like geo-location, tags, and what-not, which the server puts in an entity. In this entity the server also puts the key of a blob where the actual image data is going to be stored, and the server concludes the request by returning to the client the url returned by create_upload_url(). This works fine, however I can get inconsistency, such as if the second request is never issued, and hence the blob is never filled. The entity is now pointing to an empty blob.
The only solution to this problem I can see is to trigger a deferred task which is going to check whether the blob was ever filled with an upload. I'm not a big fan of this solution, so I'm guessing if anybody has a better solution in mind.
I went through exactly the same thought process, but in Java, and ended up using Apache Commons FileUpload. I'm not familiar with Python, but you'll just need a way of handling a multipart/form-data upload.
I upload the image and my additional fields together, using JQuery to assemble the multipart form data, which I then POST to my server.
On the server side I then take the file and write it to Google Cloud Storage using the Google Cloud Storage client library (Python link). This can be done in one chunk, or 'streamed' if it's a large file. Once it's in GCS, your App Engine app can read it using the same library, or you can serve it directly with a public URL, depending on the ACL you set.