-
Fix Amazon Aurora MySQL error 63955: Virtual index ID overflow
Your Amazon Aurora MySQL cluster looks healthy. Reads and writes succeed and every metric is green. Then a routine CREATE INDEX fails with error 63955, and from then on no index that includes a virtual column can be created on the cluster until you rebuild its databases.
Error Code: 63955.
Index cannot be created because virtual index ID limit (4294967295) was reached.
What it means: the cluster has used up the internal IDs Aurora MySQL uses when it evaluates indexes that contain a virtual column. This error does not affect existing data or ordinary indexes, but new indexes on VIRTUAL generated columns, functional indexes, and multi-valued indexes fail.
What helps short term: converting the generated column in the failing index from VIRTUAL to STORED, so that the index no longer contains a virtual column. This does not reset the IDs, and every new index that contains a virtual column still fails.
What fixes it: a logical dump and restore into a new cluster, with binary log replication to keep the cutover short.
How to get warning next time: run Aurora MySQL 3.10.0 or later, and alarm on the VIRTUAL_INDEX_ID_OVERFLOW_WARNING message in the error log.
Aurora MySQL assigns each index an internal identifier, the virtual index ID, to track index metadata for undo log management. When you create indexes through DDL operations, these operations consume the identifiers over the life of the cluster, and Aurora MySQL does not provide a way to reset them in place. Large, long-lived clusters with heavy data definition language (DDL) activity can reach the limit. Typical examples are multi-database clusters, schema-per-tenant software as a service (SaaS) platforms, and applications that continuously create, drop, and rebuild indexes.
The impact is narrow but serious. Existing tables continue serving reads and writes, this error does not cause data loss or corruption, and you can still create indexes on ordinary columns. However, CREATE INDEX, ALTER TABLE ... ADD INDEX, and CREATE TABLE fail when the new index contains a virtual column. Table-rebuilding ALTER TABLE statements on tables that already have such an index fail too. Until you rebuild the database, schema migrations and releases that add these indexes are blocked, and so is onboarding new tenants in a schema-per-tenant design whose tables use them.
In this post, we explain the limit and how to recognize error 63955. We then walk through the documented resolution, a logical dump and restore, including the binary log setup, user and grant migration, validation, cutover, and rollback limits. Finally, we set up the monitoring that turns the next occurrence into a scheduled project rather than an incident.
Which engines and versions this applies to
Error 63955 is specific to Amazon Aurora MySQL-Compatible Edition, referred to as Aurora MySQL throughout this post. The code does not exist in upstream MySQL, and the condition does not affect Amazon RDS for MySQL. The Virtual index ID overflow errors documentation says “RDS might throw error code 63955.” That wording appears because you manage Aurora clusters through the Amazon Relational Database Service (Amazon RDS) API and console. Any Aurora MySQL cluster with enough accumulated DDL can reach the limit. What differs by version is whether you get advance warning.
Starting with Aurora MySQL version 3.10.0, the engine writes a warning to the MySQL error log as the limit approaches. Earlier versions give no advance warning, so the first sign is a failed CREATE INDEX. Aurora MySQL 3.10 is also a long-term support (LTS) release. For both reasons, this post restores into 3.10 or later rather than into the source’s version.
To check your version, run the following statement, or look at the Engine version field on the cluster’s Configuration tab in the Amazon RDS console.
SELECT AURORA_VERSION();
For the authoritative description of this limit, refer to Virtual index ID overflow errors and the Aurora MySQL 3.10.0 release notes.
The virtual index ID limit
The documentation states that Aurora MySQL limits virtual index ID values to prevent an issue caused by the undo format in MySQL, and that a cluster with an index above the limit might not be available. When an index approaches the limit, or when you attempt to create one above it, Aurora MySQL returns warning or error code 63955 to the session that ran the statement.
Three points prevent the most common misreadings.
Aurora MySQL checks the limit only for indexes that contain a virtual column. Every index that the cluster creates uses up an ID, but Aurora MySQL compares the ID with the limit only when the new index includes a virtual column. Indexes on ordinary columns keep working, even on a cluster past the limit. The next section lists the index types that count.
The number in the error message is a fixed ceiling, not a reading. The client error always prints 4294967295. The error-log warning is different: it includes the ID of the index being created, so subtracting that ID from 4294967295 shows how much headroom is left. In alerting, match the error-log tags shown in Setting up the error-log alarm, not the number.
The warning is the only place the current ID is reported. No RDS API, Amazon CloudWatch metric, or Amazon CloudWatch Database Insights view shows it, so the error-log warning is your early signal.
Failing the DDL is the protective behavior. Aurora refuses the statement rather than create an index above the limit, so reads and writes on existing tables continue. Because Aurora MySQL does not provide a way to reset the identifier in place, the documented resolution is a rebuild rather than a configuration change.
Which indexes count against the limit
An index counts if it contains a virtual column, whether you define the column or MySQL creates one automatically.
A secondary index on a VIRTUAL generated column, for example c INT AS (a+b) VIRTUAL with KEY (c). A composite index counts if any one of its columns is virtual.
A functional index (MySQL 8.0.13 and later), such as KEY ((a+b)) or KEY ((LOWER(name))). MySQL builds these on a hidden virtual generated column, so they count even if you never declared a generated column.
A multi-valued index on a JSON array, such as KEY ((CAST(j->'$.tags' AS UNSIGNED ARRAY))). These also use a hidden virtual column.
Indexes on ordinary columns and on STORED generated columns still use up IDs, but Aurora MySQL does not check them against the limit.
Recognizing error 63955
Any statement that creates an index containing a virtual column fails. A functional index on a small table is enough to demonstrate it.
CREATE INDEX idx_email_lower ON app_db.customer_profile ((LOWER(email)));
Error Code: 63955. Index cannot be created because virtual index ID limit (4294967295) was reached.
The two signals surface in different places, and that decides what you can monitor.
Error 63955 is returned to the client session that ran the statement. You see it in your SQL client, your application or schema-migration tool logs, or your deployment pipeline.
The error log records a warning tagged VIRTUAL_INDEX_ID_OVERFLOW_WARNING (MY-014078) on Aurora MySQL 3.10.0 and higher, as the limit approaches. The log line does not contain 63955, but it does include the ID of the index being created. The session that ran the DDL also receives warning 63955, but the statement succeeds, and most applications and schema-migration tools do not surface SQL warnings. A cluster can therefore write the warning for weeks without anyone noticing.
In the error log, the warning and the error look like the following. The ID values are illustrative.
[Warning] [MY-014078] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_WARNING] Index `idx_email_lower` in table `app_db/customer_profile` has id 4000000123 and is approaching virtual index id limit of 4294967295.
[ERROR] [MY-014079] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_ERROR] Index `idx_email_lower` in table `app_db/customer_profile` cannot be created because index id 4294967296 is above virtual index id limit of 4294967295.
To confirm the diagnosis without touching a production table, create a functional index like the example above on a small, existing, non-critical table, and then drop it. An index on an ordinary column is not a valid test, because it succeeds even on a cluster past the limit. Use an existing table rather than creating one, because CREATE TABLE with an index that contains a virtual column can fail the same way. For retrieving the error log, refer to Monitoring Amazon Aurora log files.
Converting to STORED: A short-term workaround
Because the limit applies only to indexes that contain a virtual column, converting the generated column in a failing index from VIRTUAL to STORED and rebuilding the index takes that index out of the check. In one case, the customer could create new indexes again after this change. It is a workaround, not a fix: the conversion does not reset the IDs, and any new index that contains a virtual column, including a functional or multi-valued index, still fails. Plan the logical rebuild anyway.
Weigh these trade-offs before you use it.
The conversion rebuilds the table. Changing a column from VIRTUAL to STORED copies the table, which takes time on large tables, and the stored values add storage. The rebuild fails with error 63955 if the table has another index that still contains a virtual column.
STORED is only valid on generated columns. Applying it to an ordinary column returns Error Code: 1064, a syntax error.
Solution overview
The recommended resolution in the AWS documentation is to recreate your database with a logical dump and restore. A logical rebuild recreates the schema and reloads the data, rather than copying the existing physical structures. Snapshot restore, Aurora cloning, point-in-time recovery, and Blue/Green deployments all produce a physical copy of the volume, and none of them is a documented way to clear this condition.
The procedure has these stages:
Turn on binary logging on the source cluster.
Clone the source and record the binary log position of the clone.
Dump the clone with MyDumper and restore into a new Aurora MySQL 3.10 cluster with MyLoader.
Migrate users and grants.
Validate the new cluster.
Replicate from the source to the new cluster, starting at the recorded position.
Cut over by stopping traffic, letting replication catch up, and switching endpoints.
The documentation does not mandate a new cluster as the target. Reloading in place on the same cluster is possible, but it takes the affected databases offline for the whole reload and removes the rollback path. This post uses a new cluster.
Prerequisites
AWS Identity and Access Management (IAM) permissions to create and modify Aurora DB clusters, instances, and cluster parameter groups (rds:CreateDBCluster, rds:CreateDBInstance, rds:RestoreDBClusterToPointInTime for the clone, rds:ModifyDBCluster, rds:ModifyDBClusterParameterGroup, rds:RebootDBInstance, rds:DescribeEvents), plus logs:PutMetricFilter, logs:PutRetentionPolicy, and cloudwatch:PutMetricAlarm for the alarm.
A custom DB cluster parameter group on the source cluster. You cannot change binlog_format in the default group.
An Amazon Elastic Compute Cloud (Amazon EC2) host in the same Region and virtual private cloud (VPC) as the cluster, with MyDumper and MyLoader installed. This post was tested with MyDumper and MyLoader v1.0.11-1 against Aurora MySQL 3.10.3. Pin the version you test with.
Amazon Elastic Block Store (Amazon EBS) for the dump. Dump files are commonly about a third of the size that information_schema reports, but size the volume with headroom, because the ratio depends on your data.
A maintenance window for one reboot of the source writer, described in the next section.
Step 1: Turn on binary logging on the source
Replication from the source to the new cluster needs binary logging on the source. On Aurora MySQL, binlog_format defaults to OFF, and changing it from OFF requires a reboot of the cluster. On many clusters, this reboot is the first planned downtime of the migration, so schedule it like one. If binlog_format is already set on your source, skip to the retention step.
aws rds modify-db-cluster-parameter-group \
--db-cluster-parameter-group-name my-cluster-params \
--parameters "ParameterName=binlog_format,ParameterValue=MIXED,ApplyMethod=pending-reboot"
aws rds reboot-db-instance --db-instance-identifier my-cluster-writer
Then connect to the source writer and retain binary logs for longer than the whole dump, restore, validation, and replication window. Aurora MySQL accepts up to 2,160 hours (90 days).
CALL mysql.rds_set_configuration('binlog retention hours', 168);
CALL mysql.rds_show_configuration;
If the retention period ends before replication starts, the source deletes binary logs the new cluster still needs, and you must start the migration again. For the procedure, refer to Setting up binary log replication for Aurora MySQL.
Step 2: Clone the source and record the binary log position
Dump from an Aurora clone rather than from the source. No production workload competes with the dump, no writer-side DDL can terminate its connections, and nothing writes to the clone, so every MyDumper thread sees the same data. Create the clone after binary logging is on. For the procedure, refer to Cloning a volume for an Amazon Aurora DB cluster.
Replication must start at the source’s binary log position at the moment the clone was created. Do not use a position that MyDumper records from the clone, which describes the clone’s own binary log and would make replication skip or repeat writes. Aurora records the right position as an event on the clone’s writer instance.
aws rds describe-events \
--source-type db-instance \
--source-identifier my-cluster-clone-writer \
--duration 1440 \
--query "Events[?contains(Message, 'Binlog position')].[Date,Message]" \
--output table
The output contains a line like the following. Save the file name and position.
Binlog position from crash recovery is mysql-bin-changelog.000003 4278
The same line also appears in the clone writer’s MySQL error log.
Step 3: Dump the clone and restore into a new cluster
Create the new cluster on the current Aurora MySQL 3.10 LTS release or later. Binary log replication supports a target that is the same version as the source or newer, and the rebuild is the cheapest moment to get the version that writes the advance warning.
Before the dump, set max_execution_time to 0 in the clone’s parameter group. MyDumper SELECT statements on large tables can exceed a non-zero value, and the dump fails with Query execution was interrupted, maximum statement execution time exceeded. The parameter is dynamic. If the dump drops with Lost connection to MySQL server during query, check whether net_read_timeout, net_write_timeout, or wait_timeout have been lowered in your parameter groups, and raise them for the migration window.
Store the credentials in a defaults file rather than on the command line, so that the password does not appear in process listings.
cat > ~/.mydumper.cnf <<'EOF'
[mydumper]
user=admin
password=<password>
[myloader]
user=admin
password=<password>
EOF
chmod 600 ~/.mydumper.cnf
Dump all user databases from the clone’s cluster endpoint, excluding the system schemas.
mydumper \
--defaults-file=$HOME/.mydumper.cnf \
--host=<clone-cluster-endpoint> \
--outputdir=/mnt/full-cluster-dump \
--threads=32 \
--rows=500000 \
--compress \
--events \
--routines \
--triggers \
--sync-thread-lock-mode=SAFE_NO_LOCK \
--regex '^(?!(mysql|sys|information_schema|performance_schema)(\.|$))' \
--verbose=3
MyDumper detects Aurora and refuses to run in its default lock mode, so you must set --sync-thread-lock-mode. Without it, the dump stops with this message:
We support LOCK_ALL and SAFE_NO_LOCK modes for RDS/Aurora. Select one of them to configure --sync-thread-lock-mode
SAFE_NO_LOCK takes no global lock, and it fails the dump if the worker threads do not start at the same point in time, so a dump that completes is consistent. The --regex filter is required, not just tidy. MyDumper applies the pattern to database.table for tables and to the bare schema name for schema-level objects, such as each schema’s CREATE DATABASE statement. The (\.|$) part therefore excludes a system schema name only when a dot or the end of the name follows it. Without it, the filter also skips customer schemas whose names start with the same words, such as sys_audit or mysql_app, and the dump leaves them out without a warning. Without the filter at all, MyDumper reaches the non-transactional table mysql.general_log_backup and stops the consistent dump. A successful run logs lines like these:
Using --trx-tables options, binlog coordinates will not be accurate if you are writing to non transactional tables.
Executing in SAFE_NO_LOCK mode. This backup will fail if all threads are not in the same point in time, which ensures consistency
Transactions started, unlocking tables
Finished dump at: 2026-11-10 15:07:21
The log also reports @@tokudb_version not found - ERROR 1193. MyDumper checks for the TokuDB engine, which Aurora does not have, so you can ignore this line. The warning about --trx-tables and binlog coordinates does not affect this procedure, because replication starts from the position in the clone’s event in Step 2, not from the position MyDumper records.
Then restore into the new cluster.
myloader \
--defaults-file=$HOME/.mydumper.cnf \
--host=<new-cluster-writer-endpoint> \
--directory=/mnt/full-cluster-dump \
--threads=32 \
--verbose=3
The new cluster is empty, so MyLoader needs no option to drop or overwrite existing tables. At the end of the restore, MyLoader compares checksums of each table’s structure and indexes, the views, and the events against values recorded in the dump, and logs a line such as Structure checksum confirmed for shop.orders for each one, followed by Restore completed. These checks cover structure, not row data, so still run the validation in Step 5.
In one migration, a database that information_schema reported at about 400 GB produced a dump of about 115 GB. The reported size includes secondary index storage, which the dump carries only as DDL, plus InnoDB page overhead and fragmentation. Treat the ratio as an illustration, not a planning figure.
Step 4: Migrate users and grants
This is the step most likely to break a cutover, because it surfaces only when the application points at the new cluster and nothing can authenticate. MyDumper and MyLoader do not migrate accounts, which live in the mysql system schema.
On Aurora MySQL 3, which is compatible with MySQL 8.0, GRANT no longer creates a user, and SHOW GRANTS does not include the password. Replaying grants alone therefore fails against accounts that do not exist on the new cluster. Generate SHOW CREATE USER statements first, then SHOW GRANTS, and replay them in that order.
Run the following on the clone, not on the source. The clone matches the binary log position you recorded in Step 2. Any account created or changed on the source after the clone was taken reaches the new cluster through replication in Step 6, instead of appearing in both the script and the binary log. The query skips the RDS-reserved accounts, the AWS-managed roles, and the primary (admin) user, which already exists on the new cluster.
SET SESSION print_identified_with_as_hex = ON;
SELECT CONCAT('SHOW CREATE USER ', QUOTE(user), '@', QUOTE(host), ';')
FROM mysql.user
WHERE user NOT IN ('rdsadmin', 'rdsrepladmin', 'rdsproxyadmin', 'admin',
'mysql.sys', 'mysql.session', 'mysql.infoschema',
'rds_superuser_role')
AND user NOT LIKE 'AWS\_%'
UNION ALL
SELECT CONCAT('SHOW GRANTS FOR ', QUOTE(user), '@', QUOTE(host), ';')
FROM mysql.user
WHERE user NOT IN ('rdsadmin', 'rdsrepladmin', 'rdsproxyadmin', 'admin',
'mysql.sys', 'mysql.session', 'mysql.infoschema',
'rds_superuser_role')
AND user NOT LIKE 'AWS\_%';
Replace admin with your primary user name. print_identified_with_as_hex makes the password hashes print as hexadecimal, because caching_sha2_password hashes can contain characters that do not survive a copy and paste.
Run the generated statements, and save their output as a script with every CREATE USER statement before any GRANT. User-defined roles are accounts in mysql.user, so they come through the same way, as locked accounts with no password. The CREATE USER output for a user also carries that user’s DEFAULT ROLE, and the role it names must already exist when the statement runs. Move the CREATE USER statements for roles (the ones that end in ACCOUNT LOCK) to the top of the script. Replay the script on the new cluster after the data has loaded, because table-level and routine-level grants fail against objects that do not exist yet. Because the hashes transfer unchanged, application passwords stay the same.
On Aurora MySQL version 2, which is compatible with MySQL 5.7, the same two-statement approach works, without the print_identified_with_as_hex setting.
For a role app_read and a user app_ro that has the role as its default, the replay script looks like this (hashes shortened):
CREATE USER `app_read`@`%` IDENTIFIED WITH 'mysql_native_password' REQUIRE NONE PASSWORD EXPIRE ACCOUNT LOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT;
CREATE USER `app_ro`@`%` IDENTIFIED WITH 'mysql_native_password' AS '<hash>' DEFAULT ROLE `app_read`@`%` REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT;
GRANT USAGE ON *.* TO `app_read`@`%`;
GRANT SELECT ON `shop`.* TO `app_read`@`%`;
GRANT USAGE ON *.* TO `app_ro`@`%`;
GRANT `app_read`@`%` TO `app_ro`@`%`;
After the replay, log in to the new cluster as one or two application accounts and run a query they depend on. That confirms both the password hash and the grants before you cut over.
Step 5: Validate the new cluster
Validate more than row counts. COUNT(*) misses lost routines, triggers, events, views, and secondary indexes. Run the following on the clone and on the new cluster, and compare the output.
SELECT s.schema_name,
(SELECT COUNT(*) FROM information_schema.tables t
WHERE t.table_schema = s.schema_name AND t.table_type = 'BASE TABLE') AS tables_,
(SELECT COUNT(*) FROM information_schema.views v
WHERE v.table_schema = s.schema_name) AS views_,
(SELECT COUNT(DISTINCT st.table_name, st.index_name) FROM information_schema.statistics st
WHERE st.table_schema = s.schema_name) AS indexes_,
(SELECT COUNT(*) FROM information_schema.routines r
WHERE r.routine_schema = s.schema_name) AS routines_,
(SELECT COUNT(*) FROM information_schema.triggers tr
WHERE tr.trigger_schema = s.schema_name) AS triggers_,
(SELECT COUNT(*) FROM information_schema.events e
WHERE e.event_schema = s.schema_name) AS events_
FROM information_schema.schemata s
WHERE s.schema_name NOT IN ('mysql', 'sys', 'information_schema', 'performance_schema')
ORDER BY s.schema_name;
Then compare exact SELECT COUNT(*) results on your largest tables. For critical tables, CHECKSUM TABLE on both sides gives a stronger check, but it reads every row, so run it on the clone rather than on the source.
Finally, run the statement that failed on the old cluster. It should now succeed.
CREATE INDEX idx_email_lower ON app_db.customer_profile ((LOWER(email)));
Step 6: Replicate and cut over
Start replication on the new cluster from the source, using the binary log file name and position you recorded in Step 2. For the procedures, refer to Setting up binary log replication for Aurora MySQL, which covers mysql.rds_set_external_source and mysql.rds_start_replication.
If replication stops on an account statement, such as error 1396 (Operation CREATE USER failed), the replay script and the binary log contain the same account change. Check that you generated the script on the clone, as described in Step 4. If the account already matches the source, skip the duplicate statement with mysql.rds_skip_repl_error and replication continues.
When replication has caught up, cut over:
Stop application traffic to the source cluster.
Confirm that replication lag on the new cluster is zero, then stop replication.
Switch endpoints, using one of the options in the next section.
Resume traffic.
The dump and restore happen before this window, so the application is offline only for the switchover. Test the sequence in a non-production environment before you schedule it.
Keeping your application endpoints unchanged
Rename the cluster identifiers. Rename the old cluster, for example from my-cluster to my-cluster-old, then rename the new cluster to the original name. The cluster endpoints follow the new identifier. Instance endpoints and custom endpoints do not, so rename those separately if your applications use them. Clients also keep the old DNS response until its time to live (TTL) expires, so allow for that in the cutover window. Refer to Modifying an Amazon Aurora DB cluster.
Use an Amazon Route 53 CNAME. If applications connect through your own DNS name, update the CNAME at cutover. The same TTL consideration applies.
Use Amazon RDS Proxy. Retarget the proxy to the new cluster without changing connection strings. Refer to Using Amazon RDS Proxy.
Considerations and limitations
The binary log reboot. Turning on binary logging on the source requires a reboot, as described in Step 1.
Rollback only lasts until the first schema change. For a rollback path, configure reverse replication from the new cluster to the old one before cutover. This requires binary logging on the new cluster as well, which you can turn on before it takes traffic. The old cluster is still at the identifier limit, so any index DDL that runs on the new cluster after cutover replicates back, fails with 63955, and stops reverse replication. Freeze index DDL until you have decided not to roll back, or accept that rollback is valid only until the first schema change.
Dump the clone, not the live cluster. Replication starts from the clone’s binary log position in Step 2, so the dump must come from the same clone. A dump of the live cluster would not line up with that position.
In-place reload. Reloading on the same cluster avoids a second cluster but takes the databases offline for the whole reload and has no rollback.
Cost considerations
This migration runs paid resources in parallel. Plan for the following.
Two Aurora clusters. The source and the new cluster both bill for instance hours and storage during migration and validation.
An Aurora clone. The clone bills for its DB instances. Its storage cost starts small and grows as the clone and source diverge.
An Amazon EC2 dump host and the Amazon EBS volume that holds the dump files.
Binary log retention on the source, which adds storage for the retained logs.
Keep the validation window as short as your testing allows, and clean up as soon as validation passes.
Clean up
Most of the clean-up is ordinary deletion. Terminate the Amazon EC2 dump host, delete the Amazon EBS volume that held the dump (it is not deleted with the instance unless its DeleteOnTermination attribute is set), and delete the clone’s instances and then the clone cluster. For the procedure, refer to Deleting Aurora DB clusters and DB instances.
Three steps are specific to this migration.
Stop replication. On the new cluster, and on the old one if you configured reverse replication, run the following. The second procedure has a different name on Aurora MySQL version 2. Refer to Configuring, starting, and stopping binary log replication.
CALL mysql.rds_stop_replication;
CALL mysql.rds_reset_external_source;
Clear binary log retention on any cluster you keep that no longer replicates, so that it stops retaining log files. The value 0 is not accepted.
CALL mysql.rds_set_configuration('binlog retention hours', NULL);
Delete the old cluster last, with a final snapshot. Do this only after the new cluster has served production traffic normally and you no longer need the rollback path. The old cluster holds the only copy of your pre-migration data. Turn off deletion protection, delete its instances, then delete the cluster with a final snapshot.
aws rds modify-db-cluster --db-cluster-identifier my-cluster-old \
--no-deletion-protection --apply-immediately
aws rds delete-db-cluster --db-cluster-identifier my-cluster-old \
--final-db-snapshot-identifier my-cluster-old-final \
--no-delete-automated-backups
Prevention
Alarm on the virtual index ID overflow messages in the error log. This is your only advance signal. The next section walks through the setup.
Reduce unnecessary DDL churn. Avoid repeatedly creating and dropping indexes, and treat table-rebuilding ALTER TABLE as drawing down a finite budget.
Limit indexes that contain virtual columns on clusters with heavy DDL. Aurora MySQL does not check indexes on ordinary columns or STORED generated columns against the limit.
Plan a periodic logical rebuild for databases with very heavy DDL activity, so that the work happens on your schedule.
Stay on Aurora MySQL 3.10.0 or later. Earlier versions give no advance warning.
Setting up the error-log alarm
Turn on error-log export to Amazon CloudWatch Logs for the cluster.
aws rds modify-db-cluster \
--db-cluster-identifier my-cluster \
--cloudwatch-logs-export-configuration '{"EnableLogTypes":["error"]}'
Aurora creates the log group /aws/rds/cluster/my-cluster/error with no expiry, so set a retention period.
aws logs put-retention-policy \
--log-group-name /aws/rds/cluster/my-cluster/error \
--retention-in-days 90
Create a metric filter that matches the warning. The pattern matches the two tags that Aurora MySQL writes to the error log: VIRTUAL_INDEX_ID_OVERFLOW_WARNING as the limit approaches, and VIRTUAL_INDEX_ID_OVERFLOW_ERROR when an index is refused. Match the tags rather than the message text. Filter patterns are case sensitive, and the error-log message spells id in lowercase, unlike the client error. The bare number 63955 is also a poor match, because it can appear in log sequence numbers, thread IDs and byte counts.
aws logs put-metric-filter \
--log-group-name /aws/rds/cluster/my-cluster/error \
--filter-name aurora-virtual-index-id-warning \
--filter-pattern '?"VIRTUAL_INDEX_ID_OVERFLOW_WARNING" ?"VIRTUAL_INDEX_ID_OVERFLOW_ERROR"' \
--metric-transformations \
metricName=VirtualIndexIdWarning,metricNamespace=Custom/AuroraMySQL,metricValue=1,defaultValue=0
Test the pattern before relying on it. A healthy cluster has no warning line to test with, so the following command uses a sample line in the format that Aurora MySQL writes. The output should show one match.
aws logs test-metric-filter \
--filter-pattern '?"VIRTUAL_INDEX_ID_OVERFLOW_WARNING" ?"VIRTUAL_INDEX_ID_OVERFLOW_ERROR"' \
--log-event-messages '[Warning] [MY-014078] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_WARNING] Index `idx_b` in table `app_db/orders` has id 4000000123 and is approaching virtual index id limit of 4294967295.'
Then create the alarm. Aurora writes the error log to disk every five minutes and rotates it hourly, so a one-hour period is appropriate.
aws cloudwatch put-metric-alarm \
--alarm-name aurora-virtual-index-id-warning-my-cluster \
--alarm-description "Aurora MySQL virtual index ID overflow warning or error" \
--namespace Custom/AuroraMySQL \
--metric-name VirtualIndexIdWarning \
--statistic Sum \
--period 3600 \
--evaluation-periods 1 \
--threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:111122223333:my-dba-topic
--treat-missing-data notBreaching matters because a healthy cluster can go hours without writing to the error log. Metrics extracted from log events are billed as custom metrics.
Covering every cluster
A metric filter applies to one log group, and each cluster has its own. The following loop sets up export, retention, the filter, and the alarm for every Aurora MySQL cluster in the Region. Log export needs no parameter group change, so the loop also works on clusters that use the default parameter group.
FILTER='?"VIRTUAL_INDEX_ID_OVERFLOW_WARNING" ?"VIRTUAL_INDEX_ID_OVERFLOW_ERROR"'
TOPIC=arn:aws:sns:us-east-1:111122223333:my-dba-topic
for c in $(aws rds describe-db-clusters \
--query "DBClusters[?Engine=='aurora-mysql'].DBClusterIdentifier" --output text); do
aws rds modify-db-cluster --db-cluster-identifier "$c" \
--cloudwatch-logs-export-configuration '{"EnableLogTypes":["error"]}' >/dev/null
lg="/aws/rds/cluster/$c/error"
until aws logs describe-log-groups --log-group-name-prefix "$lg" \
--query 'logGroups[0].logGroupName' --output text | grep -q "$lg"; do sleep 30; done
aws logs put-retention-policy --log-group-name "$lg" --retention-in-days 90
aws logs put-metric-filter --log-group-name "$lg" \
--filter-name aurora-virtual-index-id-warning --filter-pattern "$FILTER" \
--metric-transformations "metricName=VirtualIndexIdWarning-$c,metricNamespace=Custom/AuroraMySQL,metricValue=1,defaultValue=0"
aws cloudwatch put-metric-alarm --alarm-name "aurora-virtual-index-id-warning-$c" \
--namespace Custom/AuroraMySQL --metric-name "VirtualIndexIdWarning-$c" \
--statistic Sum --period 3600 --evaluation-periods 1 --threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching --alarm-actions "$TOPIC"
done
The metric name includes the cluster identifier, so each alarm tracks one cluster. For an infrastructure-as-code version, the metric filter and alarm map to the AWS::Logs::MetricFilter and AWS::CloudWatch::Alarm AWS CloudFormation types.
For reference, refer to Publishing Amazon Aurora MySQL logs to Amazon CloudWatch Logs and Creating metrics from log events using filters.
Conclusion
Error 63955 means the cluster has used up the virtual index IDs Aurora MySQL uses when it evaluates indexes that contain a virtual column, including functional and multi-valued indexes. Converting a generated column to STORED is a short-term workaround for one index, no physical copy clears the condition, and the documented resolution is a logical dump and restore into a new cluster.
Plan four things carefully: the reboot to turn on binary logging, the binary log position of the clone, users and grants as their own step, and the rollback limit after the first schema change. Restore into Aurora MySQL 3.10 or later, and alarm on the VIRTUAL_INDEX_ID_OVERFLOW_WARNING message in the error log so the next cluster gives you notice.
If you are unsure which restore path fits your cluster and engine version, raise a case with AWS Support before you commit to a migration window.
For more detail, refer to Virtual index ID overflow errors, the Aurora MySQL 3.10.0 release notes, and Migrate very large databases to Amazon Aurora MySQL using MyDumper and MyLoader.
About the author
Kartik Saini
Kartik is a Cloud Support Engineer on the Amazon RDS and Amazon Aurora team at AWS. He specializes in troubleshooting MySQL-compatible database engines, and works with customers to diagnose and resolve complex Aurora MySQL issues.
-
Village News: MySQL News + Events (13 November 2026)
Welcome back to Village News, our curated roundup of MySQL and database news. This week, PostgreSQL shipped updates fixing 52 security vulnerabilities and retired PostgreSQL 14, and the FOSDEM M-Ecosystem devroom opened its call for participation.
If you want to get these updates, just subscribe to the blog.
Enjoy!
MySQL News
Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated)
Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights
Marco Tusa, Percona Blog
TL;DR - pc.weight lets you run an even-numbered Galera/PXC cluster, but working out quorum and views gets hard fast. Tusa's advice: keep to three nodes unless you truly need more.
Taming MySQL 8.4 Background Histogram Lock Wait Timeouts on Amazon RDS
Umesh Shastry, MySQL :: under_the_hood
TL;DR - On RDS for MySQL 8.4, background histogram updates collide with writes to mysql.rds_heartbeat2 and flood the log with MY-015116 warnings. The fix: drop that table's histograms, not change global settings.
Help prevent out-of-memory restarts on Aurora MySQL 8.4
AWS Database Blog
TL;DR - AWS reproduces a memory-pressure event to show how Aurora MySQL 8.4's automatic memory management avoids a restart, what to alarm on (AuroraMemoryHealthState), and when to take manual control.
How Oracle changed the future of MySQL: Dominic Preuss on Inside the Silicon Mind
VillageSQL
TL;DR - In an interview with Firas Sozan, VillageSQL's CEO covers MySQL's path from MySQL AB to Oracle, why investment went to PostgreSQL, and Oracle opening MySQL development to contributors such as AWS and Google.
DuckDB × MySQL Day: Eight Talks, One Question
OurSQL Foundation Blog
TL;DR - A walk through the agenda for the free online DuckDB × MySQL Day on November 17: eight talks on whether an embedded columnar engine can finally make analytics next to MySQL practical.
FOSDEM 2027 – The M-Ecosystem Devroom: Call for Participation
FOSDEM MySQL & Friends Devroom
TL;DR - The successor to the MariaDB, MySQL & Friends devroom runs Sunday, January 31, 2027 in Brussels. Submissions close November 29; pick "The M-ecosystem" track.
Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL
Vadim Tkachenko, Percona Blog
TL;DR - In DBTrail's published benchmark, 12 of 14 reports ran faster on its DuckDB copy than on a Percona Server replica, and the slowest fell from 225 s to 7.94 s. The two lightest reports were slower on the copy.
Amazon RDS for MySQL 26.10 is now available in Amazon RDS Database Preview Environment
AWS What's New
TL;DR - MySQL 26.10 can now be tested on RDS. It reuses read views to remove a global mutex on busy READ COMMITTED workloads and logs warnings when history-list length crosses innodb_max_purge_lag.
Diagnosing MySQL at Scale (Part 1): Critical System Blocks, Metadata Locks, and Cluster Convoys
Umesh Shastry, MySQL :: under_the_hood
TL;DR - A four-part series on reading Performance Schema waits. Part 1 covers MDL, data-dictionary latches and Group Replication convoys; Part 2 covers storage, buffer pool latches and memory spills.
Releasing iClaustron API, a new Rust NDB API with a C and a Rust API towards RonDB 26.10.0
Mikael Ronström
TL;DR - iClaustron 1.0.0 is an MIT-licensed Rust/C NDB API for RonDB that reaches 9M key lookups/s on a laptop. Separately, Ronström charted his commits: 2,320 in 2026, up from 220 in 2025.
Vector search in MySQL: from a plugin to the top of the Contributor Summit agenda
Alkin Tezuysal, AskDba
TL;DR - A history of vector search in MySQL, from the MyVector plugin through adoption by MySQL forks to native work from hyperscalers. Oracle, Google, Percona and Intel opened the Contributor Summit with vector proposals.
Database News
PostgreSQL 18.7, 17.12, 16.16, 15.20, 14.25, and 19 RC 2 Released!
PostgreSQL Global Development Group
TL;DR - The update fixes 52 security vulnerabilities, several of them arbitrary-code-execution bugs rated CVSS 8.8, and over 140 other bugs. It is the final PostgreSQL 14 release, and some fixes need post-update steps.
TLS Connect/Disconnect Performance Boost With WolfSSL
Georgi Kodinov, MariaDB Foundation Blog
TL;DR - A two-line build fix lets MariaDB's bundled WolfSSL use its __uint128_t big-number code on non-x86 64-bit platforms. TLS connect/disconnect became more than 3x faster (4.5 s to 1.3 s per 100).
From an Isolated Project to the ADBC Driver Foundry: MariaDB Joins the CDN!
Frédéric Descamps, lefred's blog
TL;DR - lefred's native ADBC driver for MariaDB, which connects it to DuckDB over Apache Arrow, is now built and distributed by Columnar's ADBC Driver Foundry, with prebuilt binaries on its CDN.
Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management
Andrey Glazkov, Percona Blog
TL;DR - Percona released an early version of PDS, a self-hosted platform for provisioning and managing databases, starting with MySQL and PostgreSQL on Kubernetes. Percona is asking for feedback.
Benchmarking MariaDB Raft Cluster vs. MariaDB Galera Cluster
Axel Schwenke, MariaDB Corporation Blog
TL;DR - With one node at 80 ms RTT, Raft Cluster in Enterprise Server 12.3 held about 2,150 TPS against Galera's 125. On a low-latency three-node cluster the two were close. See also the Raft Cluster overview.
Ten ways a glibc change can break your PostgreSQL cluster
Pablo Svampa, Percona Blog
TL;DR - glibc upgrades can change locale collation order or remove locales, which silently breaks text indexes and comparisons. The post lists ten ways this hits a Postgres cluster, including RHEL 8→9→10 locale renames.
Giving the Optimizer a Hint: What’s New in MariaDB Server 13.0, compatibility and beyond
Oleg Smirnov, MariaDB Foundation Blog
TL;DR - MariaDB 13.0 adds MySQL-style /*+ ... */ optimizer hints for join order, indexes, join buffers, derived-table merging and more. It also adds MariaDB-specific extensions for views and query blocks.
Next-Level Benchmarking Overfitting
Thomas Neumann, Database Architects
TL;DR - Neumann tests the GenDB-V paper's claimed 22–61x TPC-H speedups over Umbra and DuckDB against memory-bandwidth limits. He argues they are infeasible for real query processing and come from AI-driven benchmark overfitting.
Upcoming Database Events
pgDay México 2026
November 13, 2026
Puerta Polanco
Mexico City, Mexico
DuckDB × MySQL Day
November 17, 2026
Online
PGConf.Asia 2026
November 17-18, 2026
Hong Kong
OurSQL Foundation Webinar: What Happened to the MySQL Query Cache?
November 18, 2026, 10:00 AM ET / 16:00 CET
Online
PgConf.Nepal 2026
November 18-21, 2026
Pulchowk Campus, Tribhuvan University
Kathmandu, Nepal
PGConf.PL 2026
November 24, 2026
ARCHE Dwór Uphagena
Gdańsk, Poland
PostgreSQL Conference Japan 2026
November 27, 2026
AP Tokyo Yaesu
Tokyo, Japan
AWS re:Invent 2026
November 30 – December 4, 2026
Las Vegas, NV
OurSQL Foundation Webinar: What Is the Fastest Database? Open-Source Benchmarking with HammerDB
December 3, 2026, 10:00 AM ET / 16:00 CET
Online
Open Source Summit Japan
December 7-9, 2026
Tokyo, Japan
CIDR 2027
January 24-27, 2027
Mövenpick Hotel Amsterdam City Centre
Amsterdam, The Netherlands
MySQL Belgian Days 2027
January 28-29, 2027
ICAB Incubator
Brussels, Belgium
FOSDEM PGDay 2027
January 28-29, 2027
Brussels Marriott Hotel Grand Place
Brussels, Belgium
FOSDEM 2027
January 30-31, 2027
ULB Solbosch Campus
Brussels, Belgium
CERN PGDay 2027
February 12, 2027
CERN Council Chamber
Geneva, Switzerland
ConFoo 2027
February 24-26, 2027
Hotel Bonaventure
Montréal, QC, Canada
PGConf India 2027
March 2-5, 2027
Sheraton Grand Hotel at Brigade Gateway
Bengaluru, India
FOSSASIA PGDay 2027
March 9, 2027
Bangkok, Thailand
KubeCon + CloudNativeCon Europe 2027
March 15-18, 2027
Barcelona, Spain
Nordic PGDay 2027
March 16, 2027
Courtyard Kungsholmen
Stockholm, Sweden
SCaLE 24x
April 1-4, 2027
Pasadena Convention Center
Pasadena, CA
DuckCon #8
April 7, 2027
De Wester (Westergas)
Amsterdam, The Netherlands
PgDay Boston 2027
April 10, 2027
Tufts University Joyce Cummings Center
Boston, MA
Postgres Conference 2027
April 21-23, 2027
San Jose Marriott
San Jose, CA
Percona Live Bay Area 2027
May 11-13, 2027
Computer History Museum
Mountain View, CA
PGConf.dev 2027
May 11-14, 2027
Plaza Centre-Ville
Montréal, QC, Canada
-
Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights
We all know that when using a cluster based on Galera (Codership) such as Percona XtraDB Cluster (PXC), it is recommended to deploy an odd number of nodes.
The generic answer to this recommendation is that otherwise the risk of having an inconsistent cluster (like split-brain) is higher. (see Get started with Percona XtraDB Cluster – Percona XtraDB Cluster )
As a reminder, when split-brain occurs, a cluster will stop accepting queries, which is what a safe solution should do while waiting for the DBA to review and decide which of the two nodes is the leader. I do not even want to describe the infamous wsrep_provider_options=”pc.ignore_sb = yes”, which has been the cause of many, many nights spent trying to reconcile the corrupted datasets. If you have it, remove it.
However, in Galera, we also have the setting wsrep_provider_options=”pc.weight=N”. With it, we can define a specific weight in the quorum calculation for each node.
The question naturally arises: if we can assign a different weight to each node, why can’t we have an even number of nodes and assign them different weights?
The answer to the question is simple, yes, we can.
What is not simple is to identify who should have a different weight and how to distribute it.
To understand the problem, we need to review how the quorum is calculated (I have touched on the topic long ago here TusaCentral – Geographic replication and quorum Calculation in MySQL/Galera) and what a View is.
In Percona XtraDB Cluster (PXC) and Galera, a View (or cluster view) is the agreed-upon membership of the cluster at any specific moment in time. It represents the exact list of nodes that are currently connected, synchronized, and actively communicating with each other.
Whenever a node joins the cluster, leaves gracefully, or drops out due to a network failure, the remaining connected nodes negotiate a new View and assign it a unique View ID.
Its relationship to the Primary Component (PC) is based on quorum:
The Quorum Check: Every time a new View is formed, the nodes inside it calculate whether they hold the majority of the cluster’s total votes.
The Primary Component: If the new View holds the majority, it is designated as the Primary Component. Only the Primary Component is allowed to accept and process transactions.
Non-Primary Views: If a network split occurs, isolated nodes will form their own separate View. Because this smaller View lacks a majority, it becomes a non-primary component and immediately halts database operations to prevent split-brain data corruption.
Essentially, a View defines who is currently connected to each other, while the Primary Component is the specific View that has enough members to safely keep the database running.2026-11-10T13:45:01.498927Z 0 [Note] [MY-000000] [Galera] Node 49d8df08-bad6 state primary
2026-11-10T13:45:01.500967Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,49d8df08-bad6,93)
memb {
49d8df08-bad6,2
5ce1a9aa-8e06,2
8458c8f0-9b8f,2
}
joined {}
left {}
partitioned { 85a6e937-b8c4,2}
)The formula for calculating the quorum is:
Σ p·w < 2·(Σ m·w + Σ l·w)
Where:
p = members of the last PRIMARY view
l = members that left gracefully (LEFT)
m = members of the component being evaluated
w = pc.weight (0–255).
In short, the total weight of the previous Primary view must be less than twice the weight of the members of the new component plus the weight of the members that left gracefully. To be clear, a crashed / unreachable node (DOWN or in another partition) still counts in Σ p·w, so it pushes the threshold up.
A graceful leave (LEFT), instead, is credited to the new component. Also, any node that was not in the last PC is dropped from calculation, so a node that has just joined or rejoined contributes nothing to this check.
Look for the function:bool gcomm::pc::Proto::have_quorum(const View& view, const View& pc_view) const
{
// Compare only against members and left which were part of the pc_view.
gcomm::NodeList memb_intersection(
node_list_intersection(view.members(), pc_view.members()));
gcomm::NodeList left_intersection(
node_list_intersection(view.left(), pc_view.members()));
if (have_weights(...))
return (weighted_sum(memb_intersection, instances_) * 2
+ weighted_sum(left_intersection, instances_) >
weighted_sum(pc_view.members(), instances_));
else
return (memb_intersection.size()*2 + left_intersection.size() >
pc_view.members().size());
}(repo https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e)
Now let’s assume that we have a cluster of 4 nodes, there are many possible combinations where things will work and where instead they will hit us badly.
The most common is the split-brain scenario:
To prevent this, we may decide to use one node with different weight:
Here we go, problem solved… really?
No, because changing weights may cause other issues, for example if we calculate the weights wrongly with respect to the number of nodes:
In this case one single node could bring down the whole cluster.
Of course, if we perform standard operations, or crashes happen in cascade and the cluster has time to rebuild the Primary Component, all will work well:
The problem is that:
Cluster dimensions are dynamic and change in relation to the active Primary Component
In environments managed by operators (e.g., Kubernetes), scaling the number of nodes up or down is even more dynamic and difficult to predict.
In real life we may have an incredible variety of scenarios, I have built just a few (see PDF)
The real question is not if we can do it but whether it is worth it: does using an odd number of nodes help or not?
The answer is partly.
Odd nodes get rid of the exact-tie case in the simple situation. They don’t completely stop the cluster losing PRIMARY state.
What matters is the parity of the total weight, not the node count.
A tie happens when present = partitioned, which means the counted weight (Σw(PC) − Left) splits exactly in half.
That is only possible when the counted weight is even. So:
5 nodes of w=1, any two-way split: 3|2 or 4|1. One side always strictly wins.
6 nodes with one node at w=2 (total 7) behave exactly like an odd cluster. Adding the Galera Arbitrator (garbd) as a 7th voter does the same thing.
3 nodes weighted 2,1,1 (total 4) is an odd node count with an even total, so it can tie again.
Where odd nodes still lose PRIMARY:
The PC shrinks after each failure, and its parity changes. 5 nodes lose 1 and stay PRIMARY, but the new PC has 4 nodes. A 2|2 split after that ties. A rolling failure still ends at 2 nodes and losing one of those ties.
Graceful leaves change the parity. If 1 of 5 shuts down cleanly, the counted weight is 4, so a 2|2 split ties.
Splits into three or more groups. A 2|2|1 split of 5 has no tie, but no group is bigger than the rest combined (2 < 3), so every group goes NON-PRIMARY.
Losing a majority is still losing. 5 nodes survive 2 failures but not 3, so odd doesn’t add fault tolerance. 5 and 6 nodes both tolerate only 2 failures. The 6th node adds cost and a tie risk, and no extra resilience.
Two data centers. With 3+2 nodes, the 3-node site survives a link cut, but if that site goes down, the 2-node site stops. To survive the loss of either site we need a third location, even if it only runs garbd.
In practice: aim for an odd total weight in the PC and use a third site or an arbitrator for multi-DC setups. Even then, a cluster that has already shrunk can become even again, so the last-two-nodes tie can’t be designed away.
Only pc.ignore_sb or a manual bootstrap gets past it. But you do not want to go there.
Given this, and given that what counts is NOT the number of nodes but the balance in the weight calculation, is using weights the right way to compensate?
Well, this may be my personal opinion, but in my experience, weights make the cluster harder to manage, and the stability they add is narrow.
A weight-2 node acts like two nodes that always fail together. Four nodes weighted 2,1,1,1 behave like a 5-node cluster where two of the nodes share one box.
We trade a split-brain tie for a single point of fate:
Gain: a clean 2|2 split now has a fixed winner, the side holding the heavy node.
Loss: losing the heavy node plus one other is fatal. Over-weighting makes it worse: with w=3, losing that node alone causes a tie.
The set of survivable failures shifts rather than grows. We still tolerate only one arbitrary failure plus some specific splits.
What weights add to day-to-day operations:
Nodes are no longer equal. Everyone must know which node is “special”. Runbooks, monitoring and on-call decisions depend on it.
Maintenance order matters. Shut the heavy node down cleanly so its weight counts for neither side.
If it crashes during maintenance on another node, the cluster can stop. Patching or rebooting it may first need its weight moved to another node, a step that’s easy to forget.
Weight is a configuration that can drift. It has to be consistent across config files, rebuilds and node replacements. A rebuilt node that comes back with the default w=1 silently changes the quorum maths.
The tie-breaker site is fixed. With two data centers, the weighted side always wins. That’s no different from simply putting 3 nodes in one DC and 2 in the other, and in both cases losing that DC stops the cluster.
Shrinking still breaks it. After failures the PC changes, so a setup tuned for 4 nodes may tie at 3 or 2.
Simple to most complex for the same goal:
An odd number of equal nodes: symmetric, nothing special to manage.
An even number of nodes plus the Galera Arbitrator (garbd): the data nodes stay equal, and the arbitrator holds the tie-breaking vote. If it runs at a third site, the cluster survives losing either main site, which weights alone can’t give you.
An even number of nodes with weights: use this only when you can’t add a node or an arbitrator and you deliberately want one side to always win a split.
Weights don’t make the cluster generally more stable, and they add operational complexity. They make sense as a deliberate tiebreaker, not as a fix for even node counts.
Conclusions
I stand by the evergreen KISS motto: Keep It Simple and Straightforward.
Having a PXC cluster with more than 3 nodes makes sense in very few cases, and only to scale reads or if you plan to kill 2 nodes at the same time on a cluster of 5.
Given the significant complications the increase of nodes or the use of weight brings, I do not really see why someone in a healthy state of mind should look for trouble.
Just because it can be done, it doesn’t mean “do it”, especially if the gain is minimal and the complexity increases.
Do yourself a favor: keep your cluster to the minimum, to what is really needed and do not try to stretch it. You will save yourself from spending miserable nights doing cluster recovery or worse data reconciliation.
Great MySQL to everyone.
The post Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights appeared first on Percona.
-
MySQL Shorts - Episode 106 - MySQL LAG() Window Function for Comparing Sales Data
MySQL's LAG() window function helps you compare sales totals across rows and calculate changes between consecutive records. See how to retrieve previous row values, order sales data by date, and calculate sales differences using the LAG() function in MySQL: https://dev.mysql.com/doc/refman/26.10/en/window-function-descriptions.html#function_lag
-
Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management
It’s Thursday afternoon, and a developer has an idea.
Nothing complicated. They are working on a new application feature and want to test an idea before committing to a larger implementation. A quick prototype is already running locally. So far, so good. Next, they want to deploy the application to a shared environment so their colleagues can try it out. The application needs a database too, but it needs to be accessible from the shared environment, not just their laptop. And that’s where things get more complicated.
Now they need to submit a request to the infrastructure team, wait for an environment to be provisioned, or spend the afternoon working through Kubernetes configurations. What should have been a quick experiment becomes another task on somebody else’s backlog.
This is a common problem for engineering teams. Getting a database running in a shared environment shouldn’t require extensive coordination or infrastructure expertise. But there is another issue: when an application moves to staging or production, the requirements grow. The challenge isn’t just making provisioning easier, but finding a consistent way to manage databases as applications evolve, without requiring developers and infrastructure teams to go over the same things all over again.
Even when the underlying infrastructure is in place, creating a database with the right configuration, access controls, and operational settings can require several manual steps. And provisioning is only the beginning. Someone still needs to manage backups, monitor resource utilization, handle scaling, and maintain the database throughout its lifecycle.
At Percona, we’ve been exploring how to simplify these workflows while allowing organizations to retain control over their infrastructure. The result is Percona Database Service (PDS), a platform for provisioning and managing databases. We’re starting with MySQL and PostgreSQL on Kubernetes, and we’re making the initial implementation available for users to explore and provide feedback.
The idea is straightforward: bring the simplicity of a managed database experience to infrastructure you control, without the usual vendor lock-in.
How PDS works
PDS follows a relatively simple model: Kubernetes environments connect to a centralized control plane, allowing databases to be deployed and managed within the customer’s infrastructure.PDS builds on OpenEverest, the open-source database management platform that provides the underlying Kubernetes-based database provisioning and management capabilities. PDS extends that foundation with a centralized, more opinionated experience designed to simplify how developers interact with their databases.
Using Amazon Elastic Kubernetes Service (EKS), PDS provisions databases directly within your AWS account, rather than moving them into a Percona-hosted environment. This allows you to retain ownership of your infrastructure and databases while using PDS as a centralized management platform.
The PDS approach to database creation is intentionally opinionated. PDS provides standardized starting points for common workloads, yet still allows customization when needed. The goal isn’t to hide every technical detail. It’s to simplify and standardize routine database operations while keeping the underlying systems accessible to the teams that operate them.
PDS is currently an experimental initiative, not a generally available product. We’re using it to explore how database provisioning and management can be simplified, test different approaches with early users, and gather feedback before deciding how the platform should evolve. The capabilities described below reflect what we’re experimenting with today and may change as we learn more.
Connecting an environment is a command, not a form
It starts with a name, which doubles as the EKS cluster name, and one generated command, run locally on a machine that already has aws, eksctl, kubectl, helm, and python3. Nothing gets pasted or copied into a web form; the command does the work, and the environment reports itself in once it is up and running.
That command asks which AWS region to use. Leave it blank, and it uses your AWS CLI’s configured region, or us-west-2 if you haven’t set one. Then it creates an EKS cluster with four t3.2xlarge nodes and an S3 bucket, installs PDS onto it, and registers it. If you point the command at a cluster that’s already running PDS, it reuses that install instead of stacking a second one on top, and turns off that cluster’s own sign-in page in the process.
A database as a profile, not a spec sheet
Creating a database means picking a profile, not filling in a resource calculator: Dev, Staging, or Prod.
Each profile comes with configurations and operational defaults established by Percona’s experts, bringing best practices into the provisioning process. High availability (HA) is the default from Staging up, so it’s not a box you have to know to check. You retain control over your databases and infrastructure, while PDS helps ensure they’re configured according to proven operational standards.
Access is Private or Public. Public adds a load balancer in your own cloud account, billed by your provider, and an allowlist of IPs or CIDR ranges.
Your AI agent can drive it
It is really easy to connect an AI assistant to PDS. PDS provides setup instructions for supported clients, such as Cursor and Claude, enabling you to integrate database management into your existing workflows.
Authentication happens separately from the AI client configuration. Sign in through a browser-based flow, without storing passwords in configuration files or sharing credentials with the assistant. Once authenticated, the agent can start operating. The agent’s actions are recorded, so nothing goes unnoticed.
And then there are permissions you control. Grant read-only access to database information, allow operational tasks such as provisioning, scaling, and backups, or separately authorize actions such as deleting or restoring databases. Allowing an agent to modify databases does not automatically give it permission to delete them.
Managing databases after deployment
Creating a database is only one part of its lifecycle.
PDS provides a database management interface where users can inspect deployment status, review resource metrics, access connection information, and perform supported operational tasks. For example, users can scale database resources, configure scheduled backups, or initiate an on-demand backup. Backup schedules include configurable frequency, execution time, and retention settings. PDS also provides an activity history to help track operations performed through the platform. The interface currently exposes basic operational metrics, including CPU utilization, disk usage, and database connections.
These capabilities are intended to cover common management tasks without requiring users to interact directly with Kubernetes resources for every operation.
Supporting teams and shared environments
In most organizations, the people requesting databases are not necessarily the people responsible for operating the underlying infrastructure.
A development team may need a PostgreSQL instance for an application, while platform or DBA teams remain responsible for infrastructure, security, and operational standards. PDS supports this separation through organizations, teams, and user management. Users can register, create teams, and invite colleagues. Database environments can be organized around teams, creating a foundation for shared management and controlled access.
This is an area we’re developing based on customer feedback. One recurring requirement is to give application teams more independence without losing centralized oversight. That includes questions around permissions, resource limits, and which operations should require additional authorization.
The current implementation provides the initial team and user management capabilities. More granular governance and operational controls are areas we’re continuing to explore.
Integration with Percona Monitoring and Management (PMM)
PDS also integrates with Percona Monitoring and Management (PMM). This gives teams access to deeper database performance monitoring and diagnostics. PMM enables more detailed analysis of query performance, database activity, and potential bottlenecks for PDS users.
Together, they combine simplified database management with the monitoring capabilities Percona users already know and trust.
Try PDS and tell us what you think
We’re making PDS available to learn more about how it fits into real database workflows.
Visit pds.percona.com, create an account, and connect an Amazon EKS environment in your AWS account. From there, you can provision MySQL or PostgreSQL databases, create teams, invite users, configure backups, and explore integrations. We look forward to receiving your feedback:
Which provisioning steps create too much work?
What should a standardized configuration include?
Which operational tasks should be automated?
Where should the boundaries between developer self-service and managed database be?
This is an early preview, and some functionality will change as development continues. Our goal is to build a practical, open source approach to database management that gives teams more control over their infrastructure, not less.
Try PDS Today
The post Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management appeared first on Percona.
|
