There are many CI/CD solutions out there: http://www.devopsbookmarks.com/ci. However, looking at some buildbot examples, the snippets of Python code seem very similar to those created when writing, say, workers for RQ.
RQ seems fairly simple while Buildbot seems quite complex. Are the additional features of a full-blown CI/CD solution like Buildbot really worth it when it's possible to create queues and workers with a much simpler (yet not as fully featured) system like RQ?
In other words, what's the best way to frame the tradeoffs between CI/CD frameworks and worker queues?
We use Jenkins CI and the bonus you get with these larger frameworks are:
web interface not only for task definition, but also to review the results
wider set of plugins for different types of tasks
visualisation of test results
notification of users by e-mail
We were considering doing most of the tasks, Jenkins CI is doing for us (running tests on data) by other means (like AWS Lambda), but the visual interface is the main argument to stay with Jenkins as it allows our users to see the results without a need to do these things ourself.
Related
I have a data product that receives realtime streams of vehicles data, process the information and then displays it through rest APIS. I am looking to rebuild the ETL side of the system in order to enhance reliability and architecture order. I am considering to use Apache Airflow, but have some doubts.
The python microservices I need to orchestrate are complex and have many different dependencies, hence a monolithic solution would be huge and tricky if implemented with python operators in Airflow. Do you see any convenient options of using Airflow for these needs? Might be a better solution to use Kafka or any other Messaging System?
Thanks
Airflow is definitely used to orchestrate ETL systems however if you require real time ETL, a message queue is probably better.
However you can Trigger DAGs Externally if it suits your use case. Now depending on how distributed you want, you can also have multiple instances running to parallelize and speed up your operations.
If you think batch processing will be helpful, you can have a pseudo realtime set up with a very fast schedule. This will ( in my opinion ) be easier to debug then a traditional message queue system.
Airflow gives you a lot of modularity which lets you break down a complex architecture and debug components easily compared to a messaging queue. It also has provisions for auto requeueing and also has named queues which you can use in case you need to send large data loads to a separate processing queue on a machine with more resources.
I have worked on a similar use case as yours where we needed to process realtime data and display it via APIs and we used a mixture of Airflow and messaging queues. In my experience it was always a lot easier to figure out where and why something went wrong with Airflow due to the Airflow UI which makes getting a high level overview incredibly efficient.
I'm specifically interested in avoiding conflicts when multiple users upload (upload_file) slightly different versions of the same python file or zip contents.
It would seem this is not really a supported use case as the worker process is long-running and subject to the environment changes/additions of others.
I like the library for easy, on-demand local/remote context switching, so would appreciate any insight on what options we might have, even if it means some seamless deploy-like step for user-specific worker processes.
Usually the solution to having different user environments is to launch and destroy networks of different Dask workers/schedulers on the fly on top of some other job scheduler like Kubernetes, Marathon, or Yarn.
If you need to reuse the same set of dask workers then you could also be careful about specifying the workers= keyword consistently, but this would be error prone.
TL;DR:
Is there a possibility to easily let the workers decide which tasks they can work on, depending on their (local) configuration and the task args/kwargs?
A quick and dirty solution I thought of would be to raise Reject() in all workers that find themselves unsuitable, but I was hoping there's a more elegant one.
Details:
The application is an (educational) programming assignment evaluation tool - maybe comparable to continuous integration: A web application accepts submissions of source code for (previously specified) programming languages (or better: programming environments) which then need to be compiled and executed with several test cases. Now especially for use in a high performance computing course with GPUs, compiling and executing can not happen on the host where the web application runs (for other cases just think security reasons).
To make this easily configurable for administrators, I'd like to have a configuration file for a worker, where locally available resources, compiler types and paths etc. are configured, which the worker uses to decide whether to work on a task or not.
Simply using different queues and using a custom Router does not seem appealing to me, because the number and configuration of queues could vary at runtime and would look a little messy, I think.
Is there an elegant way to achieve something like that? To be honest, the documentation on Extensions and Bootsteps didn't give me much guidance on this.
Thanks in advance for any tips and pointers.
Question is relevant to this and this;
the difference is, I'd prefer something with possibly more precision and low load (per-minute cron job isn't preferable for those) and with minimal overhead (i.e. installing celery with rabbitmq seems like a big overkill).
An example task for such is personal reminders server (with reminders that could be edited over web and sent out through e-mail or XMPP).
I'm probably looking for something more like node.js's setTimeout but for django (and though I might prefer to implement reminders in node.js anyway, it's still a possibly interesting question).
For example, it's possible to start new threads in django app (with functions consisting of sleep() and send()); in what ways this can be bad?
The problem with using threads for this solution are the typical problems with Python threads that always drive people towards multi-process solutions instead. The problem is compounded here by the fact your thread isn't driven by the normal request-response cycle. This is summarized nicely by Malcolm Tredinnick here:
Have to disagree. Threads are not a good solution to this problem. The
issue is process management. As written, your threads will never be
rejoined. Webserver processes have a lifecycle uncontrollable by you
(the MaxRequestsPerChild Apache parameter and similar things in other
servers) and you are messing with that by using threads.
If you need a process with a lifecycle that is not matched by the
request-response path — something long running and independent of the
response — a completely separate process is definitely the right model
to use. Using a thread is tying it to the response lifecycle, which
wil have unintended side-effects.
A possible solution for you might be to have a long running process performing your tasks which gets a wake-up signal from a light cron process.
Another possibility would be build something using 0mq, which is much lighter than AMQP style queues (at the cost of some features of course). Tarek Ziade is working on a Mozilla project called powerhose that uses 0mq, looks super simple, and has a heartbeat capability with resolution to the second.
Using a combination of C, C++, Shell, Perl and/or Python I want to develop a system which allows a user to submit jobs to a queue (or queues). Jobs from the queue should be executed on one of a number of worker machines in an appropriate order.
The system should provide basic functionality. However in addition you may consider one or more of the following specific problems in more detail:
Support for multiple users
Flexible specification of jobs
Interdependencies between jobs
Integration of job creation with existing package(s)
Cancelling Jobs
Smarter scheduling of jobs
Discovery of available machines
Varying capabilities of worker nodes
Multiple Operating Systems
Failure of worker hardware
Failure of the Job Execution(including hanging indefinitely)
Failure of the queue management machine(s).
I have some basic experience in C, C++ and mostly in Python. I am interested in providing basic functionality using this system. There is mainly theory on internet about all these but I can't find examples so as to see how it works. If anyone has info, sources, example code or anything which can help me I will really apreciate it.
Why do you try to reinvent wheels?
Use http://celeryproject.org/