I am using bitnami Postgres docker image for creating my master-slave database.
I am making changes to my master and serving my app using slave.
my script needs to remove a table in order to recreate it with updated data.
But as soon as I delete these tables, they will be removed from slave as well, and hence can cause my app to break.
Is there a way to retain tables on slave until I am done recreating them?
Am I using the right approach and intuition?
Any help is much appreciated
Thanks
You can suspend replication by running this statement on the standby server:
SELECT pg_wal_replay_pause();
Then, when you are done, you can resume replication with
SELECT pg_wal_replay_resume();
Note that when you resume replication, the tables on the standby will be deleted as well. But replay will happen as fast as possible, so the time when you have no tables on the standby will be no longer than necessary.
Concerning permissions: the documentation says:
This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.
So you could
GRANT EXECUTE ON pg_wal_replay_pause() TO some_user;
Related
Say I have a SQL 2008 Database, which contains the inventory data for my business. I need to trigger a python script, once an item qty is changed. This can result under several conditions, it could be a sale order, or simply a qty change. The python script will transform the data, and upload it to google sheets.
I need this to trigger in realtime, like when the specified columns change or records are created, I need to fire off the script.
Its preferred the solution runs on the DB server itself, without having to pay for other integration tools such as Zapier. (Besides Zapier wont help here)
Constraints:
I cannot move the database to the cloud (Business Restriction)
Upgrading the database to a new version is not possible either (budget)
Changing the Database to open source is not possible either (other application dependencies)
Its a real pickle, but I'm trying to find a solution for a real time trigger.
Failing that I could almost implement a periodic scanning method, but this will create new problems.
Havent tried anything yet, because I have no idea what to try here.
Some google searches, but was not able to find a solution.
Source: https://www.sqlshack.com/use-xp-cmdshell-extended-procedure/
The answer to this is to configure the required trigger, using xp_cmdshell. You can then use xp_cmdshell to run a .bat or .py file.
Be aware of the permissions xp_cmdshell is running as, most likely this will be the SQL user that runs the file, you will have to ensure this user has the right privileges at the OS level to execute files, and write to any directories that need to be written to.
If anyone is in a similar situation, they could look into upgrading to SQL Express (which is free), not sure if this would break the application, but there is a good chance it will not(SQL Server 2008R2 Express upgrade to SQL Server 2012 Express).
It goes without saying this is certainly not best practice, and if you can at all avoid it, it would be best to run a scheduled task instead.
I'm working on a code wrote in Python 2.7 that connects to a MariaDB database to read data.
This database receives data from different external resources. My code only read it.
My service read the data once at the beginning and keep everything in memory to avoid I/O.
I would like to know if there is someway to create some 'function callback' in my code to receive some kind of alert of new update/insert, so I can reload my memory data from the database every time that any external resource change or save new data.
I have thought of creating a sql trigger to a new table to insert some "flag" there and put my service to check that new table periodically if the flag is present.
If so, reload the data and delete the flag.
But it sounds like a wrong workaround...
I'm using:
Python 2.7
MariaDB Ver 15.1 Distrib 10.3.24-MariaDB
lib mysql-connector 2.1.6
The better solution for MariaDB is streaming with the CDC API: https://mariadb.com/resources/blog/how-to-stream-change-data-through-mariadb-maxscale-using-cdc-api/
The plan you have now, with using a flag table, means your client has to poll the flag table for presence of the flag. You have to run a query against that table at intervals, and keep doing it 24/7. Depending on how quickly your client needs to be notified of a change, you might need to run this type of polling query very frequently, which puts a burden on the MariaDB server just to respond to the polling queries, even when there is no change to report.
The CDC solution is better because the client can just request to be notified the next time a change occurs, then the client waits. It does not put extra load on the MariaDB server, any more than if you had simply added a replica server.
Currently trying to use pg_dump and pg_restore to be able to dump select rows from a production server to a testing server. The goal is to have a testing server and database that contains the subset of data selected, moreover through a python script, I want the ability to restore the database that original subset after testing and potentially modifying the contents of the database.
From my understanding of pg_dump and pg_restore, the databases that they interact with must be of the same dbname. Moreover, a selection criteria should be made with a the COPY command. Hence, my idea is to have 2 databases in my production server, one with the large set of data and one with the selected set. Then, name the smaller set db 'test' and restore it to the 'test' db in the test server.
Is there a better way to do this considering I don't want to keep the secondary db in my production server and will need to potentially make changes to the selected subset in the future.
From my understanding of pg_dump and pg_restore, the databases that they interact with must be of the same dbname.
The databases being worked with only have to have the same name if you are using --create. Otherwise each programs operates in whatever database was specified when it was invoked, which can be different.
The rest of your question is too vague to be addressable. Maybe pg_dump/pg_restore are the wrong tools for this, and just using COPY...TO and COPY...FROM would be more suitable.
I am working on scaling out a webapp and providing some database redundancy for protection against failures and to keep the servers up when updates are needed. The app is still in development, so I have chosen a simple multi-master redundancy with two separate database servers to try and achieve this. Each server will have the Django code and host its own database, and the databases should be as closely mirrored as possible (updated within a few seconds).
I am trying to figure out how to set up the multi-master (master-master) replication between databases with Django and MySQL. There is a lot of documentation about setting it up with MySQL only (using various configurations), but I cannot find any for making this work from the Django side of things.
From what I understand, I need to approach this by adding two database entries in the Django settings (one for each master) and then write a database router that will specify which database to read from and which to write from. In this scenario, both databases should accept both reads and writes, and writes/updates should be mirrored over to the other database. The logic in the router could simply use a round-robin technique to decide which database to use. From there on, further configuration to set up the actual replication should be done through MySQL configuration.
Does this approach sound correct, and does anyone have any experience with getting this to work?
Your idea of the router is great! I would add that you need automatically detect whether a databases is [slow] down. You can detect that by the response time and by connection/read/write errors. And if this happens then you exclude this database from your round-robin list for a while, trying to connect back to it every now and then to detect if the databases is alive.
In other words the round-robin list grows and shrinks dynamically depending on the health status of your database machines.
The another important notice is that luckily you don't need to maintain this round-robin list common to all the web servers. Each web server can store its own copy of the round-robin list and its own state of inclusion and exclusion of databases into this list. This is just because a database server can be seen from one web server and can be not seen from another one due to local network problems.
I am building a small system which throws data from a mongodb collection, it already works fine but I have to restart it everytime I make changes.
I already have a monitor that dectect changes and restarts the server automatically but I want to do something like this with mongodb changes.
I am currenlty using CentOs 5, Nginx, uWsgi & python2.7.
I'd look into using tailable cursors, which remain alive after they've reached the end of a collection, and can block until a new object is available.
Using PyMongo, you can call Collection.find with a tailable=True option to enable this behavior. This blog post gives some good examples of its usage.
Additionally, instead of just querying the collection, which will only alert you to new objects added to that collection, you may want to query the database's oplog, which is a collection of all insert, updates, and deletes called on any collection in the database. Note that replication must be enabled for mongo to keep an oplog. Check out this blog post for info about the oplog and enabling replication.