Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
Are Airflow and Nifi perform the same job on workflows? What are the pro/con for each one?
I need to read some json files, add more custom metadata to it and put it in a Kafka queue to be processed. I was able to do it in Nifi. I am still working on Airflow. I am trying to choose the best workflow engine for my project
Thank you!
For a great overview of Airflow and Apache NiFi checkout this reddit post: https://www.reddit.com/r/bigdata/comments/51mgk6/comparing_airbnb_airflow_and_apache_nifi/
For your specific use-case of ingesting Json files, enriching them and routing them to Kafka I believe NiFi is the right tool for the job. A couple of processors you could potentially use, as well as documentation for each, are below:
GetFile: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.GetFile/index.html
JoltTransformJSON: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.JoltTransformJSON/index.html
PublishKafka (or PublishKafka_0_10 depending on your version): https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.9.2/org.apache.nifi.processors.kafka.pubsub.PublishKafka/index.html
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 days ago.
Improve this question
Goodnight.
I'm developing a project that is a Web System for managing a company in which it will register customers and leads through an excel file, files that will contain 20k, 30k and even 60k. And I'm using FastAPI for the Back End.
I would like ideas on the best way to do this registration:
Use a Webhook: The API receives the file, copies the 60k data and sends it in JSON to a Webhook that will register in PostgreSQL.
Using the API: The API performs all registrations asynchronously, the user sends it through an asynchronous request using JavaScript to the API and it takes care of the rest, which can break the API?
Which option do you think is better? Share ideas with me.
At the moment I'm waiting for ideas to execute.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
Should I use flask or Quart or just crossbar to serve dynamic and asynchronous web page ? I need websocket with RPC and PubSub.
I usually work on database with sqlalchemy , and I like brython instead of javascript.
I'd like to avoid Node.js if possible.
You can find a list of implementations here
I'm planning to use Node.js and the Crossbar.io documentation is very helpful. Seems like a good choice.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I would like to create a Flask Python server to serve for a Mobile App / Web server as visualization for Data from PLC. I do not know how to connect Flask Server with another script and to exchange Data between it in full duplex. Is it possible to do it over Sqlite ? Can I write / read a data to the same DB from 2 scripts / servers ? I draw some explanation of my problem (first picture is a problem and a second one may be the possible solution ?/ thanks in advance for your experienced suggestions)enter image description here
Create apis in flask and use requests module from another script and consume the apis that are built on Flask.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have a working R script. Can this be wrapped in a Python code so it can be deployed as an API ?
As mentioned in an earlier post, things that are not easy in R can be relatively simple in other languages. Another example would be connecting to Amazon Web Services. In relation to s3, although there are a number of existing packages, many of them seem to be deprecated, premature or platform-dependent. (I consider the cloudyr project looks promising though.)
If there isn’t a comprehensive R-way of doing something yet, it may be necessary to create it from scratch. Actually there are some options to do so by using AWS Command Line Interface, AWS REST API or wrapping functionality of another language.
http://jaehyeon-kim.github.io/2015/11/Quick-Test-to-Wrap-Python-in-R.html
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I've been asked to develop a video review system.
it would need multiple levels of permissions e.g. internal review, client review.
notes(comments) on all the submissions.
revisions of the video assets.
searching of video assets.
publishing these video assets will need to be done from the commandline. because it would be as a final step in a long chain of task being handed to our render-farm.
initially we don't need to video's to play in the browser directly just have a link to the source file and to a generated mov.
Is Django the right thing to use for something like this? is it difficult to create "content" revision control? how is django with commandline input? or is there something better to do this?
any thoughts would be appreciated.
Lars
Django will do this just fine. There are a coulpe revision control apps out there, a quick google search turned up this: Django-Revision as far as command-line usage you can write your own management commands.