-
Group Replication Beyond a Single Cluster: DC-DR with Percona (PS MySQL) Operator
A while ago, we discussed the cross-site replication feature of the Percona PXC operator. Recently, a similar cross-site replication feature was introduced in the Percona (PS MySQL) operator v1.2.0, a topology based on Group Replication/InnoDB Cluster.
In this blog post, we will explore how to add a DR Cluster to an existing DC Cluster to form a ClusterSet environment, which provides a seamless switchover between DC/DR members. The good part is that all the complexity and configuration will be managed by the Percona operator, with very few setup steps required.
Group Replication (DC-DR) using Innodb ClusterSet
Let’s discuss how we can implement a cross-site replication (DC-DR) topology across two separate database clusters.
Environment used for Demonstration
Two separate database clusters (pscluster1 and pscluster2) are deployed within a single GCP/Kubernetes environment, each managed by a dedicated operator.
Percona Operator for MySQL( based on Percona Server for MySQL) has a version. 1.2.0
MySQL version 8.4.10.
Network connectivity between DC/DR components.
DC (pscluster1) configuration
Step 1: Deploying the cluster on the DC side.shell> kubectl get ps -n ps
NAME REPLICATION ENDPOINT STATE MYSQL ORCHESTRATOR HAPROXY ROUTER AGE
ps-cluster1 group-replication ps-cluster1-haproxy.ps ready 3 3 28mshell> kubectl get pods -n ps
NAME READY STATUS RESTARTS AGE
percona-server-mysql-operator-6b78d5f68c-bzg6r 1/1 Running 0 30m
ps-cluster1-haproxy-0 2/2 Running 0 28m
ps-cluster1-haproxy-1 2/2 Running 0 27m
ps-cluster1-haproxy-2 2/2 Running 0 27m
ps-cluster1-mysql-0 2/2 Running 0 29m
ps-cluster1-mysql-1 2/2 Running 0 28m
ps-cluster1-mysql-2 2/2 Running 0 26mStep 2: Retrieve the DC’s InnoDB cluster name, which will be required later to construct the ClusterSet.shell> kubectl get ps ps-cluster1 -n ps -o jsonpath='{.status.innodbClusterName}{"\n"}'
pscluster1Step 3: Get a service endpoint which is used when setting up the ClusterSet environment. We will use ps-cluster1-mysql-primary, which is mapped to the current Primary node in the existing cluster.shell> kubectl get services -n ps
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ps-cluster1-haproxy ClusterIP 34.118.225.215 <none> 3306/TCP,3307/TCP,3309/TCP,33060/TCP,33062/TCP 29m
ps-cluster1-mysql ClusterIP None <none> 3306/TCP,33062/TCP,33060/TCP,6450/TCP,33061/TCP 29m
ps-cluster1-mysql-primary ClusterIP 34.118.229.137 <none> 3306/TCP,33062/TCP,33060/TCP,6450/TCP,33061/TCP 29m
ps-cluster1-mysql-proxy ClusterIP None <none> 3306/TCP,33062/TCP,33060/TCP,6450/TCP,33061/TCP 29m
ps-cluster1-mysql-unready ClusterIP None <none> 3306/TCP,33062/TCP,33060/TCP,6450/TCP,33061/TCP 29mStep 4: The DR site should have the same credentials as DC. We need to export the DC secret containing the credentials and import it into the DR cluster.shell> kubectl get secret ps-cluster1-secrets -n ps -o yaml > source-secret.yamlWe also need to perform a couple of cleanups in the exported secret file.
Remove the annotations, creationTimestamp, resourceVersion, selfLink, and uid metadata fields.
As per the requirement, change the namespace/instance and other unwanted details as per your replica site.
E.g,shell> yq eval 'del(.metadata.ownerReferences, .metadata.annotations, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.selfLink, .metadata.uid)' source-secret.yaml > replica-secret.yamlshell> yq eval '.metadata.namespace = "ps-dr"' -i replica-secret.yamlshell> sed -i '' 's/ps-cluster1/ps-cluster2/g' replica-secret.yamlThe final ready secret file looks like this. apiVersion: v1
data:
clusterset: XVpMQz0+LEVDe3kyQGR6TQ==
heartbeat: YmFONW1KTE8oLTxdVVB7Kg==
monitor: UC1oOHokRksoMnpSMWsmJg==
operator: SDMrblpSQm0yRHdhV3dteX0=
orchestrator: cEVqMEUlaGp5fShudyZURHtl
replication: ZjldZlkwa3ZwNWU8KVpsag==
root: e3BycEl+LlcrPFVUUTwpaG4=
xtrabackup: eFhNNENIRF4wJXZ7bjxbKi11ag==
kind: Secret
metadata:
labels:
app.kubernetes.io/component: database
app.kubernetes.io/instance: ps-cluster2
app.kubernetes.io/managed-by: percona-server-mysql-operator
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: percona-server
name: ps-cluster2-secrets
namespace: ps-dr
type: Opaque
DR (pscluster2) Cluster configuration
Step 1: First, we need to import the modified secret file replica-secret.yaml and initialise the DR cluster.shell> kubectl apply -f deploy/replica-secret.yaml -n ps-dr
secret/ps-cluster2-secrets createdshell> kubectl get secrets -n ps-dr
NAME TYPE DATA AGE
...
ps-cluster2-secrets Opaque 8 12s
...Also, we need to make some modifications to the custom resource file cr.yaml to initialise the DR cluster.metadata:
finalizers:
- percona.com/delete-mysql-pods-in-order
# - percona.com/delete-ssl
# - percona.com/delete-mysql-pvc
name: ps-cluster2
spec:
crVersion: 1.2.0
secretsName: ps-cluster2-secretsmysql:
clusterType: group-replication
size: 3
image: percona/percona-server:8.4.10-10.1
bootstrap:
mode: manualshell> kubectl apply -f deploy/cr.yaml -n psNote: We set spec.mysql.bootstrap.mode to manual so Pod mysql-0 does not form a Group Replication group until the ClusterSet adopts it and references the secret file we copied from the DC.
It is also expected that after applying the Custom Resource file cr.yaml, the Pod mysql-0 starts but stays NotReady. The cluster ps-cluster2 reports the Initialising state and the AwaitingExternalBootstrap condition in status. Pods mysql-1 and mysql-2 do not start until Pod mysql-0 joins the Group Replication.
Once we deploy the custom resource, we can notice the status below. Please note that the complete Pods will be ready only once DR successfully syncs and joins the DC cluster. shell> kubectl get ps -n ps-dr
NAME REPLICATION ENDPOINT STATE MYSQL ORCHESTRATOR HAPROXY ROUTER AGE
ps-cluster2 group-replication ps-cluster2-haproxy.ps-dr initializing 17mshell> kubectl get pods -n ps-dr
NAME READY STATUS RESTARTS AGE
percona-server-mysql-operator-6b78d5f68c-4gsm9 1/1 Running 0 28m
ps-cluster2-mysql-0Step 2: We need to note down the DR InnoDB cluster name, which will later be used by the ClusterSet.shell> kubectl get ps ps-cluster2 -n ps -o jsonpath='{.status.innodbClusterName}{"\n"}'
pscluster2Step 3: We need to identify the DR cluster endpoint that is reachable from the DC. This endpoint will be used to feed the clone and perform other management operations.
We will use the local FQDN ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local of the ps-cluster2-mysql-0 Pod, which is currently initialised and awaiting the ready state.shell> kubectl get pods -n ps-dr
NAME READY STATUS RESTARTS AGE
percona-server-mysql-operator-6b78d5f68c-4gsm9 1/1 Running 0 48m
ps-cluster2-mysql-0 1/2 Running 0 37mBy connecting to the DC Primary Pod, we can verify cross-communication between the DC and the DR.shell> kubectl exec -it ps-cluster1-mysql-0 -n ps -- shsh-5.1$ curl -v ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local:3306
* Trying 10.60.1.7:3306...
* Connected to ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local (10.60.1.7) port 3306 (#0)
> GET / HTTP/1.1
> Host: ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local:3306
> User-Agent: curl/7.76.1
> Accept: */*
Data Restoration Part
By default, when the DR site initialises and attempts to join the ClusterSet, it receives data from the source/DC via the clone recovery method.
The Operator uses MySQL Shell to create a physical snapshot of the dataset from the Source/DC and transfer it to the DR Replica.
Alternatively, we can manually perform a backup and restore on the target cluster. This strategy helps when integrating the DR site into the ClusterSet, as pre-populating the data and GTID history allows it to apply only delta changes rather than performing a full initial synchronisation. For large datasets or to prevent excessive load and performance degradation on the donor node, a backup-and-restore approach can be considered.
Here, we will use the clone recovery method to sync the DR cluster.
Next, we will deploy the ClusterSet configurations so that DR joins the DC cluster via a data clone and connects via Asynchronous Replication to stay in sync.
Below is the clusterset.yaml file where we pass the various information like (Secrets, DC/DR Endpoints, Cluster Name etc) , which we fetched in some of the above steps earlier.kind: PerconaServerMySQLClusterSet
metadata:
name: my-cluster-set
finalizers:
- percona.com/clusterset-dissolve
spec:
# unsafeFlags:
# forcedFailover: false
# forcedClusterRemoval: false
primaryCluster: pscluster1
credentialsSecret:
name: ps-cluster1-secrets
key: clusterset
sslMode: AUTO
createReplicaClusterOptions:
recoveryMethod: clone
clusters:
- innodbClusterName: pscluster1
endpoints:
- host: ps-cluster1-mysql-primary.ps.svc.cluster.local
port: 3306
- innodbClusterName: pscluster2
endpoints:
- host: ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local
port: 3306
mysqlshellRunner:
image: percona/percona-server:8.4.10-10.1We will apply the changes only to the DC node.kubectl apply -f deploy/clusterset.yamlPlease note: Member “ps-cluster2-mysql-0” on the DR side will connect to the DC via asynchronous replication. While the local members of the DR will join via the Group Replication mechanism.
This action will also start a separate backend job on DC, which creates an associated Pod to perform the ClusterSet activity.shell> kubectl get jobs -n ps
NAME STATUS COMPLETIONS DURATION AGE
my-cluster-set-pscluster2-add-replica Complete 1/1 23s 53s….shell> kubectl get pods -n ps
NAME READY STATUS RESTARTS AGE
my-cluster-set-pscluster2-add-replica-wk7xd 1/1 Running 0 3sAs soon as the process completes successfully, the job and associated Pod will be removed from the list.
There is one extra pod that can be noticed, and it persists. This provides a utility/client for running MySQL Shell commands against the MySQL ClusterSet.shell> kubectl get pods -n ps
NAME READY STATUS RESTARTS AGE
my-cluster-set-runner-5d68fbcc74-svmpw 1/1 Running 0 10hIf the job fails or errors occur, we need to investigate the exact problem using the information below.kubectl describe job <jobname>
kubectl describe pod <add-replica pod>
kubectl logs <add-replica pod>Finally, we can check the Pod status on the DR cluster. It will now reflect all MySQL and haproxy pods in the fully completed/ready state. shell> kubectl get pods -n ps-dr
NAME READY STATUS RESTARTS AGE
percona-server-mysql-operator-6b78d5f68c-4gsm9 1/1 Running 0 11h
ps-cluster2-haproxy-0 2/2 Running 0 36m
ps-cluster2-haproxy-1 2/2 Running 0 36m
ps-cluster2-haproxy-2 2/2 Running 0 36m
ps-cluster2-mysql-0 2/2 Running 1 (37m ago) 11h
ps-cluster2-mysql-1 2/2 Running 1 (36m ago) 36m
ps-cluster2-mysql-2 2/2 Running 1 (34m ago) 35m
Verification of ClusterSet completion
We can check the ClusterSet information below if it has been processed successfully without any errors.shell> kubectl get ps-clusterset my-cluster-set -n ps
NAME PRIMARY ENDPOINT READY
my-cluster-set pscluster1 ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306 Trueshell> kubectl get ps-clusterset my-cluster-set -n ps -o yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLClusterSet
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"ps.percona.com/v1","kind":"PerconaServerMySQLClusterSet","metadata":{"annotations":{},"finalizers":["percona.com/clusterset-dissolve"],"name":"my-cluster-set","namespace":"ps"},"spec":{"clusters":[{"endpoints":[{"host":"ps-cluster1-mysql-primary.ps.svc.cluster.local","port":3306}],"innodbClusterName":"pscluster1"},{"endpoints":[{"host":"ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local","port":3306}],"innodbClusterName":"pscluster2"}],"createReplicaClusterOptions":{"recoveryMethod":"clone"},"credentialsSecret":{"key":"clusterset","name":"ps-cluster1-secrets"},"mysqlshellRunner":{"image":"percona/percona-server:8.4.10-10.1"},"primaryCluster":"pscluster1","sslMode":"AUTO"}}
creationTimestamp: "2026-09-11T05:20:08Z"
finalizers:
- percona.com/clusterset-dissolve
generation: 5
name: my-cluster-set
namespace: ps
resourceVersion: "1789143099049519007"
uid: 6cfdce40-e1e0-47c8-bdae-1cd847938f29
spec:
clusters:
- endpoints:
- host: ps-cluster1-mysql-primary.ps.svc.cluster.local
port: 3306
innodbClusterName: pscluster1
- endpoints:
- host: ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local
port: 3306
innodbClusterName: pscluster2
createReplicaClusterOptions:
recoveryMethod: clone
credentialsSecret:
key: clusterset
name: ps-cluster1-secrets
mysqlshellRunner:
image: percona/percona-server:8.4.10-10.1
primaryCluster: pscluster1
sslMode: AUTO
status:
clusters:
pscluster1:
clusterRole: PRIMARY
globalStatus: OK
primary: ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306
pscluster2:
clusterRole: REPLICA
globalStatus: OK
primary: ""
conditions:
- lastTransitionTime: "2026-09-11T05:25:07Z"
message: ""
reason: DeploymentReady
status: "True"
type: MySQLShellRunnerReady
- lastTransitionTime: "2026-09-11T05:25:10Z"
message: ""
reason: ClusterSetBootstrapped
status: "True"
type: ClusterSetBootstrapped
- lastTransitionTime: "2026-09-11T05:25:13Z"
message: All Clusters available.
reason: ClusterSetHealthy
status: "True"
type: Ready
lastObservedAt: "2026-09-11T16:11:39Z"
lastObservedGeneration: 5
primaryCluster: pscluster1
primaryClusterEndpoint: ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306Also, we can manually access any running MySQL Pod and confirm the ClusterSet status.shell> kubectl exec -it ps-cluster1-mysql-0 -n ps -- shsh-5.1$ mysqlsh --uri=root@localhost:3306MySQL localhost:3306 ssl JS > var clusterset = dba.getClusterSet()
MySQL localhost:3306 ssl JS > clusterset.status({extended:1})
{
"clusters": {
"pscluster1": {
"clusterRole": "PRIMARY",
"globalStatus": "OK",
"primary": "ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306": {
"address": "ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306",
"memberRole": "PRIMARY",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE",
"version": "8.4.10"
},
"ps-cluster1-mysql-1.ps-cluster1-mysql.ps:3306": {
"address": "ps-cluster1-mysql-1.ps-cluster1-mysql.ps:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLagFromImmediateSource": "",
"replicationLagFromOriginalSource": "",
"role": "HA",
"status": "ONLINE",
"version": "8.4.10"
},
"ps-cluster1-mysql-2.ps-cluster1-mysql.ps:3306": {
"address": "ps-cluster1-mysql-2.ps-cluster1-mysql.ps:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLagFromImmediateSource": "",
"replicationLagFromOriginalSource": "",
"role": "HA",
"status": "ONLINE",
"version": "8.4.10"
}
},
"transactionSet": "50cc4081-ad9a-11f1-b815-1602eb3926cb:1-4,65edbac6-ad9a-11f1-9dbe-1602eb3926cb:1-139"
},
"pscluster2": {
"clusterRole": "REPLICA",
"clusterSetReplication": {
"applierStatus": "APPLIED_ALL",
"applierThreadState": "Waiting for an event from Coordinator",
"applierWorkerThreads": 4,
"receiver": "ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr:3306",
"receiverStatus": "ON",
"receiverThreadState": "Waiting for source to send event",
"replicationSsl": "TLS_AES_128_GCM_SHA256 TLSv1.3",
"replicationSslMode": "REQUIRED",
"source": "ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306"
},
"clusterSetReplicationStatus": "OK",
"globalStatus": "OK",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr:3306": {
"address": "ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr:3306",
"memberRole": "PRIMARY",
"mode": "R/O",
"readReplicas": {},
"replicationLagFromImmediateSource": "",
"replicationLagFromOriginalSource": "",
"role": "HA",
"status": "ONLINE",
"version": "8.4.10"
},
"ps-cluster2-mysql-1.ps-cluster2-mysql.ps-dr:3306": {
"address": "ps-cluster2-mysql-1.ps-cluster2-mysql.ps-dr:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLagFromImmediateSource": "",
"replicationLagFromOriginalSource": "",
"role": "HA",
"status": "ONLINE",
"version": "8.4.10"
},
"ps-cluster2-mysql-2.ps-cluster2-mysql.ps-dr:3306": {
"address": "ps-cluster2-mysql-2.ps-cluster2-mysql.ps-dr:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLagFromImmediateSource": "",
"replicationLagFromOriginalSource": "",
"role": "HA",
"status": "ONLINE",
"version": "8.4.10"
}
},
"transactionSet": "50cc4081-ad9a-11f1-b815-1602eb3926cb:1-4,65edbac6-ad9a-11f1-9dbe-1602eb3926cb:1-139",
"transactionSetConsistencyStatus": "OK",
"transactionSetErrantGtidSet": "",
"transactionSetMissingGtidSet": ""
}
},
"domainName": "my-cluster-set",
"globalPrimaryInstance": "ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306",
"metadataServer": "ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306",
"primaryCluster": "pscluster1",
"status": "HEALTHY",
"statusText": "All Clusters available."
}
Validate Replication
Log in to the Primary member and perform some writes.
shell> export PRIMARY=$(kubectl get pods -n ps -l mysql.percona.com/primary=true -o jsonpath='{.items[0].metadata.name}')
shell> echo $PRIMARY;
ps-cluster1-mysql-0
shell> kubectl exec -it ps-cluster1-mysql-0 -n ps -- sh
shell> mysql -uroot -pmysql> CREATE DATABASE IF NOT EXISTS test;
mysql> CREATE TABLE IF NOT EXISTS test.t1 (id INT PRIMARY KEY); INSERT INTO test.t1 VALUES (1);
Connect to the DR and verify the sync.
shell> kubectl exec -it ps-cluster2-mysql-0 -n ps-dr -- sh
shell> mysql -uroot -pmysql> SELECT * FROM test.t1;
+----+
| id |
+----+
| 1 |
+----+We can also visit any MySQL Pod and run the following command to get the Primary member and group replication details.mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+------------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+------------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 50cc4081-ad9a-11f1-b815-1602eb3926cb | ps-cluster1-mysql-0.ps-cluster1-mysql.ps | 3306 | ONLINE | PRIMARY | 8.4.10 | MySQL |
| group_replication_applier | 87137873-ad9a-11f1-ae3f-36b4b899b0ff | ps-cluster1-mysql-1.ps-cluster1-mysql.ps | 3306 | ONLINE | SECONDARY | 8.4.10 | MySQL |
| group_replication_applier | b952f0d3-ad9a-11f1-95a5-d6e7036e522a | ps-cluster1-mysql-2.ps-cluster1-mysql.ps | 3306 | ONLINE | SECONDARY | 8.4.10 | MySQL |
+---------------------------+--------------------------------------+------------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)
DC-DR switchover/failover
Performing a planned switchover or an ad hoc failover process is quite simple here. All we need to execute the operations below.
Switchover:
shell> kubectl patch ps-clusterset my-cluster-set -n $SOURCE_NS \
--type=merge -p '{"spec":{"primaryCluster":"replicacluster"}}'Forced Failover:shell> kubectl patch ps-clusterset my-cluster-set -n $SOURCE_NS --type=merge -p '{
"spec": {
"primaryCluster": "replicacluster",
"unsafeFlags": {
"forcedFailover": true
}
}
}'So let’s try a Primary switchover activity from DC pscluster1 to DR pscluster2. Currently, pscluster2 has a REPLICA role.MySQL localhost:3306 ssl JS > clusterset.status()
{
"clusters": {
"pscluster1": {
"clusterRole": "PRIMARY",
"globalStatus": "OK",
"primary": "ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306"
},
"pscluster2": {
"clusterRole": "REPLICA",
"clusterSetReplicationStatus": "OK",
"globalStatus": "OK"
}
},
"domainName": "my-cluster-set",
"globalPrimaryInstance": "ps-cluster1-mysql-0.ps-cluster1-mysql.ps:3306",
"primaryCluster": "pscluster1",
"status": "HEALTHY",
"statusText": "All Clusters available."
}
Run Switchover command:
shell> kubectl patch ps-clusterset my-cluster-set -n ps \
--type=merge -p '{"spec":{"primaryCluster":"pscluster2"}}'
perconaservermysqlclusterset.ps.percona.com/my-cluster-set patched
Watch the progress:
shell> kubectl get ps-clusterset my-cluster-set -n ps -w
shell> kubectl get jobs -n ps | grep switchoverAfter some time, we can see pscluster2 become the Primary cluster.shell> kubectl get ps-clusterset my-cluster-set -n ps -w
NAME PRIMARY ENDPOINT NAME PRIMARY ENDPOINT READY
my-cluster-set pscluster1 False
my-cluster-set pscluster2 ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr:3306 Truekubectl exec -it ps-cluster1-mysql-0 -n ps -- sh
sh-5.1$ mysqlsh --uri=root@localhost:3306MySQL localhost:3306 ssl JS > clusterset.status()
{
"clusters": {
"pscluster1": {
"clusterRole": "REPLICA",
"clusterSetReplicationStatus": "OK",
"globalStatus": "OK"
},
"pscluster2": {
"clusterRole": "PRIMARY",
"globalStatus": "OK",
"primary": "ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr:3306"
}
},
"domainName": "my-cluster-set",
"globalPrimaryInstance": "ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr:3306",
"primaryCluster": "pscluster2",
"status": "HEALTHY",
"statusText": "All Clusters available."
}Once the switchover finishes successfully, all new writes now go to the new Primary cluster ps-cluster2 and the old DC “ps-cluster1” will automatically become the Async Replica. The application should connect with a load balancer (Haproxy/MySQL Router) endpoint, e.g., (ps-cluster2-haproxy), which will forward all requests to the backend node (by default) to the Primary member.shell> kubectl get svc ps-cluster2-haproxy -n ps-dr
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ps-cluster2-haproxy ClusterIP 34.118.226.22 <none> 3306/TCP,3307/TCP,3309/TCP,33060/TCP,33062/TCP 12h
Key Takeaways
Configuring and managing a multi-region topology manually or on virtual machines can be quite challenging. By packaging these components within the Percona PS Operator/K8S, setting up cross-region environments becomes far simpler. This streamlined deployment is critical for establishing robust disaster recovery solutions, offloading production workloads, or scaling read operations. Furthermore, built-in support for seamless DC-DR switchovers and ad hoc failovers adds significant value to the architecture.
Before deploying these topologies into production environments, it is strongly advised to thoroughly test and validate their behaviour in lower or non-production environments. Proceed with production deployment only after achieving full confidence in the setup.
The post Group Replication Beyond a Single Cluster: DC-DR with Percona (PS MySQL) Operator appeared first on Percona.
-
Too many GCache Page Files in MySQL Data Directory
A few thousand gcache.page.* files in a Percona XtraDB Cluster (PXC) data directory is not something you see every day. We came across a case where these files had been accumulating over time and slowly consuming disk space. So, let’s dig into what happened.
At first glance, it looked like GCache had simply stopped cleaning itself up. The investigation started by answering two simple questions: when did the files start appearing, and what changed in the cluster at that time?
Finding the starting point
The oldest files showed the issue started on July 9.[hostx] percona@hostx: ~ $ ls -lh /var/lib/mysql/mysql-data/gcache.page.*
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000000
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000001
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000002
...
-rw-r----- 1 mysql mysql 128M Jul 26 07:45 /var/lib/mysql/mysql-data/gcache.page.005976
-rw-r----- 1 mysql mysql 128M Jul 26 07:50 /var/lib/mysql/mysql-data/gcache.page.005977
-rw-r----- 1 mysql mysql 128M Jul 26 07:56 /var/lib/mysql/mysql-data/gcache.page.005978The newest files showed they stopped being created on July 26, which immediately provided a timeline to investigate.
The creation of GCache page files wasn’t random. It started at a specific point in time and stopped after the next MySQL restart.
Can large transactions cause this?
Normally, you don’t see thousands of GCache page files unless Galera cannot reclaim old pages or an exceptionally large writeset forces additional page allocation. In this environment, the GCache ring file was around 60 GB, making the large writeset theory very unlikely – actually Impossible! Because there’s a hard limit of the largest transaction size at 2GB.
That pushed the investigation toward the error log, where Galera was found logging:2026-07-09T21:41:04.617132Z 91699 [Note] [MY-000000] [Galera] Freezing gcache purge at 16198595494
2026-07-09T21:41:04.622051Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/mysql-data/gcache.page.000000 of size 134217728 bytesThis was the first strong clue. When gcache.freeze_purge_at_seqno is active, Galera stops reclaiming old GCache pages. As replication continues, new page files are allocated while existing ones remain on disk.
Correlating with cluster activity
Looking a few seconds later in the error log revealed a cluster partition. The timing is difficult to ignore: Galera froze GCache purging, created the first page file, and then recorded the membership change.2026-07-09T21:41:04.617132Z 91699 [Note] [MY-000000] [Galera] Freezing gcache purge at 16198595494
2026-07-09T21:41:04.622051Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/mysql-data/gcache.page.000000 of size 134217728 bytes
2026-07-09T21:41:14.537019Z 53033 [Note] [MY-010559] [Repl] Multi-threaded replica statistics for channel '': seconds elapsed = 122; events assigned = 110401704; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 7659954362300 waited (count) when Workers occupied = 47189196 waited when Workers occupied = 22964040282100
2026-07-09T21:42:02.139966Z 91770 [Note] [MY-010914] [Server] Aborted connection 91770 to db: 'unconnected' user: 'percona' host: '10.1.163.3' (Got an error reading communication packets).
2026-07-09T21:42:17.061513Z 0 [Note] [MY-000000] [Galera] forgetting c3acab8a-8a74 (ssl://10.1.21.5:4567)
2026-07-09T21:42:17.061596Z 0 [Note] [MY-000000] [Galera] Node 7e3c8f1e-ad75 state primary
2026-07-09T21:42:17.061615Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,7e3c8f1e-ad75,11)
memb {
7e3c8f1e-ad75,1
}
joined {
}
left {
}
partitioned {
c3acab8a-8a74,1
}
)Although the logs do not explicitly state why purge was frozen, the sequence of events strongly suggests that Galera retained the writesets so the partitioned node could potentially perform an IST when it rejoined.
How could the purge have been frozen?
Further investigation into the codebase and documentation hinted that it is practically impossible that Galera can actually invoke the gcache pages purge pause and only practical way to do it is using:SET GLOBAL wsrep_provider_options='gcache.freeze_purge_at_seqno=XYZ';Reference: https://github.com/percona/galera/pull/132
Related reading: No SST node rejoins in PXC
Reproducing the behavior
Rather than stopping with a theory, Peter Sylvester (SoS) reproduced the behavior in the lab by manually setting gcache.freeze_purge_at_seqno and generating workload with Sysbench. The result matched what we observed in the production.
Additionally, even though gcache.keep_pages_count=3, Galera continued creating additional page files because purging was frozen.mysql> show global status like 'wsrep_last_committed';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| wsrep_last_committed | 94559 |
+----------------------+-------+
1 row in set (0.00 sec)
mysql> SET GLOBAL wsrep_provider_options="gcache.freeze_purge_at_seqno=94559";
Query OK, 0 rows affected (0.00 sec)Sysbench was then run on the cluster’s source host to generate logs…[root@CENTOS9-1 ~]# sysbench oltp_read_write --db-driver=mysql --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --mysql-port=3306 --table_size=1000 --tables=4 --threads=4 --rand-type=uniform --range_size=100 --time=0 --rate=0 --report_interval=5 run
WARNING: Both event and time limits are disabled, running an endless test
sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 4
Report intermediate results every 5 second(s)
Initializing random number generator from current time
Initializing worker threads...
Threads started!
[ 5s ] thds: 4 tps: 311.80 qps: 6246.62 (r/w/o: 4373.81/1248.40/624.40) lat (ms,95%): 16.71 err/s: 0.00 reconn/s: 0.00Following page files are present in datadir[root@CENTOS9-3 ~]# ls -lh /var/lib/mysql/*cache*
-rw-r----- 1 mysql mysql 11M Aug 6 14:14 /var/lib/mysql/galera.cache
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000000
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000001
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000002
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000003
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000004
-rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000005
-rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000006After the load completed, the mysqld was restarted to observe if the page files were then cleared! But they were not.[root@CENTOS9-3 ~]# systemctl stop mysql
[root@CENTOS9-3 ~]# systemctl start mysql
[root@CENTOS9-3 ~]# ls -lh /var/lib/mysql/*cache*
-rw-r----- 1 mysql mysql 11M Aug 6 14:17 /var/lib/mysql/galera.cache
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000000
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000001
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000002
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000003
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000004
-rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000005
-rw-r----- 1 mysql mysql 10M Aug 6 14:17 /var/lib/mysql/gcache.page.000006
Another interesting observation
Later the MySQL was restarted, expecting the startup to reclaim the unused pages. It didn’t. Every page file remained on disk after a restart. Even after the explicit configuration of gcache.freeze_purge_at_seqno=-1.
This suggests that startup recovery does not automatically remove these accumulated page files simply because gcache.freeze_purge_at_seqno has been cleared. At least in testing, once purge has been frozen and page files accumulate, restarting MySQL alone is not enough to reclaim the space.
We have a new bug in place for getting this behaviour sorted: PXC-5323
Can the files be deleted?
To answer that, MySQL was stopped, the local GCache files were removed and the node was started again.
Galera recreated the required cache structures automatically. More importantly, the node successfully completed an IST; and in our tests, deleting the local page files did not force an SST.
Before removing the files, make sure the node is stopped and the cluster has another healthy node that can provide the required writesets. Always validate this behavior in your own environment before using it operationally. Our testing showed that an IST was sufficient.
Conclusion
Based on both the production logs and our lab testing, the accumulation of gcache.page.* files was caused by GCache purging being frozen. Restarting MySQL did not reclaim the accumulated files in our testing. The behavior where a PXC node continuously creates new GCache page files is already fixed under: PXC-4495.
The practical workaround is to stop MySQL, remove the local galera.cache and gcache.page.* files, and start the node again. In our testing, the node rejoined the cluster using IST without requiring a full SST. As always, make sure another healthy node has the required writesets before performing this cleanup.
The post Too many GCache Page Files in MySQL Data Directory appeared first on Percona.
-
Traceability Matters: Gopal Shankar on Opening Up MySQL Development for the Next Decade
“Would making this capability available help a meaningful part of the MySQL community build better applications or run better systems?”
Q1. MySQL has recently made available several features from the enterprise tier to the MySQL Community Edition. What is driving this direction?
The simple answer is that Community Edition needs to stay strong for the people who build, run, and depend on MySQL every day. That includes developers, DBAs, startups, large enterprises, and software vendors.
When we bring broadly useful capabilities into Community Edition, particularly in areas like observability, high availability, performance, and developer experience, the whole MySQL ecosystem benefits. Users get a better platform, and we get better feedback from real workloads at scale.
Recent examples include replication observability and Group Replication capabilities, OpenTelemetry support, the Hypergraph Optimizer, Profile-Guided Optimization, and enhanced JSON Duality View support in Community Edition.
I see this as a clear sign of Oracle’s long-term commitment to MySQL. The Community Edition is fundamental to MySQL. Enterprise Edition continues to address additional commercial support, security, and operational needs, but the success of Community Edition is essential to the success of MySQL overall.
Q2. When Oracle makes a feature available in Community Edition, how are those decisions made?
There is not a single checklist that applies to every feature. We start with a practical question: would making this capability available help a meaningful part of the MySQL community build better applications or run better systems?
From there, we look at technical maturity, operational impact, security, compatibility, documentation, and long-term maintainability. A feature has to work well not only in a carefully controlled environment, but also in the many different environments where MySQL is deployed.
The strongest opportunities are often in areas that improve the daily experience of using MySQL: understanding what is happening in the server, operating highly available systems, diagnosing performance problems, and reducing unnecessary complexity for developers. We will continue to evaluate those opportunities as MySQL evolves.
Q3. What does MySQL do better than competitors in community engagement, and where can it improve?
MySQL’s strength is the combination of a mature open source database, deep engineering investment, and an ecosystem that has been built over decades. MySQL runs important workloads for organizations of every size, so reliability, compatibility, upgrades, tooling, and operational simplicity matter deeply to our community.
We also see opportunities to improve. The best open source communities make it easy for people to understand how an idea moves from discussion to action. We want to improve that path in MySQL through clearer design discussions, better issue triage, more visible contribution paths, and more predictable review and feedback.
Code is important, but it is not the only meaningful contribution. Testing, documentation, benchmarking, production feedback, tools, and community education all make MySQL better. We want contributors to feel that these forms of participation are recognized and useful.
Q4. MySQL 9.7.0 LTS brings capabilities previously limited to MySQL Enterprise Edition—including JSON Duality Views, the Hypergraph Optimizer, and improvements to replication observability and HA behavior—into MySQL Community Edition. Which of these changes do you think will have the greatest real-world impact for DBAs and developers, and what should teams consider before adopting them in production?
I would separate immediate operational impact from longer-term developer impact. For DBAs running Group Replication, the replication observability and HA changes will probably have the fastest and broadest benefit. Better visibility into flow control, applier lag and throughput, unhealthy members, and primary election helps teams diagnose problems earlier and make failover behavior more predictable. This is practical day-to-day value, especially for teams operating clusters at scale.
For developers, JSON Duality Views may be the more consequential change over time. They let teams work with JSON documents while retaining relational integrity and a single source of truth. The Hypergraph Optimizer can also be significant for complex queries, but its benefit will vary more by workload.
Teams should approach an LTS upgrade with thorough validation. Test representative workloads, failure scenarios, replication behavior, upgrades, and monitoring integrations in staging first. For the Hypergraph Optimizer, compare plans and performance for important queries. For JSON Duality Views, validate the data model, update paths, permissions, and concurrency behavior. And for telemetry, make sure the collector, retention policy, and handling of potentially sensitive operational data are ready before turning it on in production. The Community Edition additions cover replication and HA behavior, telemetry, JSON Duality Views, and the Hypergraph Optimizer.
Q5. You were personally involved in designing JSON Duality Views. What problem does it solve?
JSON Duality Views solve a problem many application teams know well. Developers often prefer JSON because it maps naturally to APIs and application objects. But relational modeling gives them normalization, transactional consistency, referential integrity, and SQL.
Historically, teams often had to choose one model, or build and maintain their own mapping layer between application objects and relational tables. In some cases, they also ended up duplicating data across multiple systems.
JSON Duality Views let an application work with hierarchical JSON documents while the underlying data remains relational. The application can use the model that feels natural for the task, but MySQL still provides a single source of truth.
For a team, that can mean less mapping code and simpler synchronization. It does not remove the need for good schema or API design, and it will not fit every application, but it gives suitable workloads a much simpler way to combine document-style development with relational strengths.
Q6. How do you balance new capabilities with MySQL’s simplicity and reliability?
MySQL has earned trust because it is practical. People can deploy it, operate it, upgrade it, and troubleshoot it with confidence. New capabilities must preserve that experience.
We pay close attention to defaults, configuration, backward compatibility, documentation, and operational behavior. Early Access builds, LTS releases, compatibility testing, and upgrade guidance are the practical mechanisms that help us validate that balance before broad adoption. The MySQL 9.6 foreign-key work is a good example. We moved foreign-key checks and cascades into the SQL layer so that those changes are visible to binary logs and CDC tools, while preserving compatibility, validating performance, and providing a temporary `innodb_native_foreign_keys` fallback for staged adoption. A feature should be powerful when users need it, while preserving the straightforward core MySQL experience for everyone else.
Not every user needs every new capability. Success is giving developers and DBAs useful new options while maintaining the stable, predictable MySQL experience that existing users rely on.
Q7. What does the more open community model look like in practice?
For a developer or DBA, participation should not begin only when they have a patch ready. They can discuss roadmap topics, share use cases, test Early Access releases, file actionable bugs, join GitHub discussions, contribute documentation or benchmarks, and participate in community events and contributor summits.
The important change is connecting these activities more clearly. If someone raises a good issue or proposal, they should be able to see where it goes next. Does it become a design discussion? A bug investigation? A request for testing? A roadmap input? That traceability matters.
Over the coming releases and community cycles, the community should see clearer guidance, more public technical discussion, improved GitHub workflows, and more structured ways to engage early. What matters most is whether people find the process easier to use and receive useful follow-up.
Q8. What changes are being made to improve the contributor experience, and how will you measure success?
The first improvement is clarity. Contributors need to know where to start, what information is needed, how review works, and what happens if a proposal is not accepted as submitted.
We are working toward clearer templates, better-defined contribution paths, more visible technical discussions, and stronger links between issues, proposals, patches, and bug records. This should make it easier for contributors and users to follow the progress of an idea.
The second improvement is feedback. When a contribution needs further refinement, people should receive a clear outcome, and where possible, practical guidance about what to do next. We will look at evidence: response and resolution trends, time to initial triage, review cycle time, contributor growth, contribution quality, roadmap participation, Early Access adoption, and feedback from contributors. We also intend to share progress regularly, pairing timely acknowledgment with meaningful follow-through.The goal is not simply to collect more pull requests. It is to create a community process that produces better outcomes.
Q9. MySQL recently marked 30 years. What will define the next decade?
MySQL must remain the practical, dependable choice for the application workloads that matter most: transactional systems, cloud-native services, distributed applications, and data-intensive workloads.
That means continued investment in performance, high availability, observability, security, developer productivity, and operational simplicity. These may not always be the most visible areas of innovation, but they are the reasons people trust a database in production.
The other important aspect is community participation. MySQL cannot thrive for another decade based only on work from one company. The community can influence priorities earlier, contribute effectively, build tools and extensions, share operational knowledge, and see that its feedback leads to visible action.
By the time MySQL turns 40, I would like it to be known not only for scale and reliability, but also for a community that has a real and practical role in shaping its future.
Qx. Anything else you wish to add?
I would encourage people to engage with MySQL early and directly. Try the Early Access releases, share concrete production experience, bring specific use cases, and tell us where the friction is.
The most useful feedback is grounded in real workloads and comes with enough detail for us to act on it. MySQL has always evolved through the combined work of engineers, users, customers, partners, and contributors. We want the next chapter to be even more collaborative.
………………………………………………………..
Gopal Shankar, Director of MySQL Engineering, Oracle.For over 20 years, I have worked at the heart of database engine architecture. Currently, as the Director of MySQL Engineering, I lead the organization responsible for the strategy, development, and roadmap of one of the world’s most popular database platforms. My expertise lies in the core internals of MySQL specifically kernel-level development, performance tuning, and scalability. I believe in solving complex technical challenges by prioritizing architectural simplicity and resilience. I have led the design of several important features, including the MySQL 8.0 Data Dictionary and Information Schema, as well as the recent JSON Duality feature. Additionally, I have helped architect the integration of foreign key handling directly into the SQL layer, effectively resolving long-standing trigger cascade limitations in MySQL recently. My goal is to deliver features that are high-performing, reliable and developer-friendly. I am focusing towards executing the MySQL roadmap and ensuring MySQL platform remains a powerful, solid foundation for modern applications. Beyond strategy, I stay connected to the kernel-level complexities tackling issues like high CPU usage, database corruption, and throughput bottlenecks. I am focused on fostering technical excellence and delivering a database engine that evolves with the needs of the industry.https://www.linkedin.com/in/gopal-shankar-1b34664/
………………….
Follow us on X
Follow us on LinkedIn
-
Vibe Coding a Database Lab: How I Built DBCanvas to Stop Rebuilding the Same Test Environment
Percona gives us room to work on our own AI-assisted projects, and I used mine to fix a problem I kept running into. Every time I wanted to try a new database feature, debug something tricky, or reproduce a customer issue, I ended up rebuilding much of the same infrastructure: DNS, TLS, Docker networking, database topologies, users and test data. Over the years, I wrote scripts to automate this but I still found myself copying and pasting post-installation steps from one lab to the next.
That repetition is what turned into DBCanvas, a self-hosted lab for designing, deploying, operating, and stress-testing multi-node database stacks on my own machine. Just design a topology on a canvas, click Deploy, and get real running nodes connected to the services and supporting infrastructure your test requires. Then, use the tools built into it or third party tools to load those databases, watch them work, and figure out why they’re misbehaving.
The code is up at github.com/jaimesicam/dbcanvas.
Figure 1: Deploying a multi-node MySQL topology with monitoring and orchestration
It’s vibe coded, and that’s the point
I want to be upfront about this that DBCanvas is vibe coded and built conversationally with an AI coding assistant rather than hand-written line by line. That turned out to be exactly the right approach for a tool whose whole job is to remove setup friction.
My initial loop or workflow looked like this. I started by asking the assistant to generate UI/UX demos for the frontend with React, a backend with Go, a drag-and-drop node canvas, a user management system and I iterated it until it felt right. Once I was satisfied, I asked it to turn those requirements into a SCAFFOLD.md which contained a full blueprint precise enough for the coding agent to rebuild the app from scratch as it contained the tech stack, naming conventions, directory tree, backend behavior, frontend behavior and the interactive details of the node editor itself.
Figure 2: The initial interface prototype that established the visual direction
Figure 3: The initial node-editor prototype for composing connected services
From there, I added features incrementally, budgeted by whatever tokens I had available in a session and logged every change in an IMPLEMENTATION.md so that I have a record of what it took to go from the original scaffold to wherever the project currently stood.
Figure 4: IMPLEMENTATION.md records each feature added after the initial scaffold
If I ever needed to rebuild the project from nothing, those two files are essentially the whole story. I know it is crude, but it’s my first time building with this many moving parts. It has held up so far at least for me.
The current loop looks like this:
Hit friction while testing, debugging, or learning something new.
Describe the environment that would remove that friction.
Let the assistant scaffold the automation: versions, configuration, identity, data, and tooling for that environment.
Keep whatever turned out to be reusable inside DBCanvas for next time.
Turn the whole workflow into something you can drive from a browser.
When Percona Server 8.4.11-11 shipped OpenID Connect authentication, I didn’t want to manually set up a Keycloak instance, wire up realms and clients, create sample identities, and configure Percona Server’s OIDC plugin every time I wanted to poke at it. Instead, that became a new DBCanvas setup. Deploy Keycloak + Percona Server + sample identities with a few clicks. You can inspect the generated OIDC configuration, authenticate with an ID token, and verify the mapped MySQL role, all without setting this up yourself. Vibe coding is what made it fast enough to build that scaffolding the same day the feature landed, and DBCanvas turned it into a reusable setup I can deploy again whenever I need it.
Figure 5: A Percona Server OIDC lab with Keycloak and a success mapped-role login
It’s not just about one feature… it’s a whole lab
OIDC with Keycloak is just one example. DBCanvas can also build a broader range of database environments:
MySQL: Percona XtraDB Cluster, Percona Server, MySQL Community, asynchronous replication, InnoDB Cluster and Group Replication, and MariaDB
PostgreSQL: standalone, Patroni, repmgr, Spock multi-master, CloudNativePG or Crunchy PGO on Kubernetes
MongoDB: Percona Server for MongoDB as standalone, replica set, or sharded cluster
Valkey: standalone and cluster
And around them, the infrastructure that makes a stack behave like a real environment. An Intranet node can provide DNS, mail, OpenLDAP, a Squid proxy and a certificate authority. Other nodes provide PMM, ProxySQL, HAProxy, Orchestrator, SeaweedFS S3, Keycloak, OpenBao, Samba AD DC, and a Kubernetes frame that runs any of six database operators.
A useful lab also needs activity. DBCanvas ships application simulators for a hotel booking system, an airline, a car rental fleet, and a stock exchange. Spin up a MongoDB replica set and point the stock market simulator at it and writes begin flowing through the set. You can observe elections and inspect real oplog activity instead of manually generating load against an idle cluster.
Figure 6: A MongoDB replica set running the stock market simulator with diagnostic tools attached
Leaning on tools other people already built
DBCanvas can make deployments more useful by integrating tools my colleagues have built for managing, troubleshooting and analyzing database environments. I can add them as part of the deployment workflow and place it alongside the database it is intended to work with.
MClusterAdmin, a MongoDB administration panel created by Przemek Malkowski, runs as its own node alongside a MongoDB deployment. It displays topology and replica-set status, sharding and the balancer, slow queries with explain, indexes, users and roles, all in a browser tab.
Figure 7: MClusterAdmin displaying the replica-set state for a DBCanvas deployment
Big Hole, an FTDC viewer, developed by Zelmar Michelini that decodes diagnostic.data. Drag in a folder and Big Hole charts every metric it can find. In DBCanvas, collecting the data is just as straightforward: right-click a MongoDB node, open the context menu, and download a compressed archive containing its logs and diagnostic data. Simply decompress the archive to your download directory, then drag the folder into Big Hole to visualize FTDC metrics alongside related log events.
Figure 8: Big Hole visualizing FTDC metrics downloaded from a MongoDB node
The same principle applies to deeper diagnostic work. DBCanvas automates the setup around proven tools instead of replacing them.
For example, the Operator Debugger uses Delve to step through Kubernetes operators with breakpoints, call stacks and variables.
Figure 9: Operator Debugger paused at a breakpoint inside a Percona operator
There’s also the Core Dump Analyzer where you can mount a core dump and matching binaries read-only, then inspect it with GDB.
Figure 10: Web-based Core Dump Analyzer with equivalent terminal command for manual troubleshooting
Ease of Use
While DBCanvas makes it easy to deploy environments but for troubleshooting, you still need to look deeper into the implementation within the nodes. Accessing the deployment via web terminal, regular terminal or web-based Filemanager would be helpful to have command of the deployment.
Figure 11: The file manager inspecting a MongoDB configuration inside a deployed node
The node menu provides the exact Docker exec command for direct access from a regular terminal.
Figure 12: Direct container access from a Docker exec command copied from the node menu
Testing beyond deployment
DBCanvas also comes with a data generator, a parallel query runner, a benchmark tool (OLTP/OLAP, read-write and read-only), and a packet inspector that decodes MySQL, PostgreSQL, MongoDB and Valkey traffic off the wire.
Figure 13: Data Generator creating sample rows for a deployed database
Figure 14: Packet Inspector decoding MongoDB traffic from the lab network
Together, these tools help reproduce problems facing a real deployment as I can generate data, apply load, inspect queries and examine network traffic.
Try it
bash
Copy
Copied!
git clone https://github.com/jaimesicam/dbcanvas.git && cd dbcanvas
make install
That builds the node images and starts DBCanvas at http://localhost:8080. The first run takes a while since it’s building docker images from scratch and learning which software versions are available for each OS.
Three important caveats:
DBCanvas creates disposable labs for previewing, testing and learning. It’s not a production software. It uses default credentials and favors setup speed over production security.
My workstation remains its primary test environment. If you run it elsewhere and encounter a bug or rough edge, please open a GitHub issue.
Some features are still maturing. I chose to bring many capabilities into one project, which means a few areas still need additional refinement and testing. For example, the Core Dump analyzer could benefit from more sophisticated variable extraction as well as the automatic detection of the appropriate operating system and compatible debug libraries to deploy.
If you’ve ever rebuilt the same test cluster for the third time this month, or wished you could hand a colleague a working reproduction instead of a page of setup instructions, try DBCanvas as it might save you some of the time it has saved me.
-
NodeJS MySQL Select Unique
I ran SELECT UNIQUE age FROM users on a table with a duplicated age. It returned 22, 17 and 15, the same rows SELECT DISTINCT returns. On MySQL 8 that statement stops before the table is read, because UNIQUE never entered the server’s select grammar, and DISTINCT is the modifier both servers accept. Which unique […]
|
