www.bortolotto.eu

Newsfeeds
Planet MySQL
Planet MySQL - https://planet.mysql.com

  • November 2026 MySQL Contributor Summit: Advancing Community Collaboration 
    On November 4–5, the MySQL community gathered online for the third MySQL Contributor Summit of 2026. Across two days, contributors shared proposals, demonstrated progress, and discussed how to make MySQL easier to extend, operate, and contribute to.  The summit marked another step in the community engagement plan announced in January. Progress since then includes new […]

  • Announcing VillageSQL Server 0.0.8
    VillageSQL Server 0.0.8 is now available. This release is about using extensions from applications: through prepared statements, with text in other character sets, and with the server protected when an extension function throws an exception. Highlights of this release include the following: Functions that take custom types work in prepared statements. Extensions built with the 0.0.8 SDK can tell raw bytes from text. When a function in an extension built with the 0.0.8 SDK throws an exception, only that statement fails, and the server keeps running. Custom types in prepared statements Many applications keep values out of the SQL text. They send the query once as a prepared statement and pass the values separately, and most MySQL connectors can work this way. In 0.0.8, functions that take custom types work in prepared statements: mysql> PREPARE re FROM 'SELECT complex_real(?) AS re'; mysql> SET @v = '(1.5,-2.5)'; mysql> EXECUTE re USING @v; +------+ | re | +------+ | 1.5 | +------+ Vector search works the same way: a nearest-neighbor query with its search value passed as a parameter uses the vector index. Custom indexes are still a preview feature, behind vsql_allow_preview_extensions=ON. Running a prepared statement many times also costs less. For many extension functions, the server no longer prepares a statement that calls them again at every EXECUTE. For a prepared nearest-neighbor query, that repeat preparation took about 10% of the server's time. Text and raw bytes A password hash is raw bytes, and an email address is text. MySQL keeps the two apart, and extensions built with the 0.0.8 SDK can too. A hash reaches the extension byte for byte, and text reaches it in one encoding, whatever character set your client uses. vsql-crypto 0.0.8 uses the new BINARY type. Its hashes come back as binary data, ready to store in a VARBINARY column: mysql> SELECT CHARSET(digest('hello', 'sha256')) AS charset; +---------+ | charset | +---------+ | binary | +---------+ mysql> CREATE TABLE hashes (id INT PRIMARY KEY, h VARBINARY(32)); mysql> INSERT INTO hashes VALUES (1, digest('hello', 'sha256')); mysql> SELECT id, HEX(h) FROM hashes; +----+------------------------------------------------------------------+ | id | HEX(h) | +----+------------------------------------------------------------------+ | 1 | 2CF24DBA5FB0A30E26E83B2AC5B9E29E1B161E5C1FA7425E73043362938B9824 | +----+------------------------------------------------------------------+ Custom types read text in other character sets too. Here the COMPLEX type from an example extension reads values from a ucs2 column: mysql> CREATE TABLE src (id INT PRIMARY KEY, s VARCHAR(40) CHARACTER SET ucs2); mysql> INSERT INTO src VALUES (2, '(3.0,4.0)'); mysql> CREATE TABLE points (id INT PRIMARY KEY, c COMPLEX); mysql> INSERT INTO points SELECT id, s FROM src; Query OK, 1 row affected (0.00 sec) mysql> SELECT id, c FROM points; +----+-------+ | id | c | +----+-------+ | 2 | (3,4) | +----+-------+ When an extension throws an exception Extensions run inside the server process. When a function in an extension built with the 0.0.8 SDK throws a C++ exception, VillageSQL ends that one statement with an error, and the server keeps running. This test extension throws on purpose: mysql> SELECT vsql_exceptions_test.exception_add(10, 20); ERROR 3200 (HY000): VDF error in function 'exception_add': VDF threw an exception (vsql_exceptions_test::exception_add) mysql> SELECT 'server still up' AS status; +-----------------+ | status | +-----------------+ | server still up | +-----------------+ Crash-safe extension installs If the server crashes in the middle of INSTALL EXTENSION, it does not leave a half-written extension behind. If an extension file is cut short, INSTALL EXTENSION rejects it. And if an extension update fails at restart, the server rolls back only that extension. For extension authors These features come from protocol 5 of the interface between the server and its extensions, which is the stable protocol in 0.0.8. If you built an extension with the development SDK of 0.0.4 through 0.0.7, rebuild it with the stable 0.0.8 SDK. Function arguments and results can be BINARY, raw bytes that the server never converts, and STRING arguments always arrive as utf8mb4. A function's prerun check can see the values of constant arguments, including a value bound to a ? in a prepared statement, so a function can accept a NULL literal without accepting every string. A C++ exception that escapes a function becomes a SQL error on the statement. A custom type can register a from_binary function beside from_string, to accept values in a binary form, such as a vector of packed floats. The server uses it for a _binary string, a VARBINARY column, or a hex literal. A function like tvector_dim(?) takes its type parameters from the value bound at EXECUTE. Upgrading from 0.0.7 0.0.8 does not load extensions built for protocol 4, so if your 0.0.7 server has one installed, the server will not start after you upgrade. To check whether any of your installed extensions use protocol 4, run this query on your 0.0.7 server: SELECT EXTENSION_NAME FROM INFORMATION_SCHEMA.EXTENSION_REGISTRATION WHERE NEGOTIATED_PROTOCOL = 4; To upgrade, dump your data with mysqldump, install your extensions on a new 0.0.8 data directory, and then load the dump. Bundled extensions The install script, the Docker image, and the release tarballs include these extensions with VillageSQL Server 0.0.8. To use one, run INSTALL EXTENSION with its name. vsql-ai 0.0.8 vsql-boolean 0.0.3 vsql-crypto 0.0.8 vsql-cube 0.0.8 vsql-fuzzystrmatch 0.0.2 vsql-http 0.0.4 vsql-network-address 0.0.8 vsql-oauth2 0.0.5 vsql-rest 0.0.11 vsql-trgm 0.0.5 vsql-uuid 0.0.9 You can find a list of all available extensions here. To build a new extension or port an existing PostgreSQL extension, you can use an agent skill. Special thanks Thanks to @kash2104, who made the server print <error decoding value> for a custom value it cannot decode, and cleaned up how the server reads raw custom values. Thanks to @jobala, whose work lets an extension function's prerun check tell a NULL literal from other values, and to @askdba, who added a MyVector profile to vector-dev-bench, our tool for measuring in-database vector indexes. There's more in this release than fits here. The release notes carry the per-PR detail. Our roadmap is published on GitHub, and we'd like to hear what you're building and what we should work on next, either on GitHub or on Discord. If you want to support VillageSQL, please star the repo.

  • Modernize on Your Terms: How Nutanix and AWS Accelerate Migrations, Simplify Operations, and Enable AI
    Learn how Nutanix and AWS help organizations migrate workloads without refactoring, simplify hybrid cloud operations, and build a flexible foundation for AI.

  • Why MySQL OR is slower than IN
    Queries with OR conditions are commonly used to implement business logic. However, it might be slower than we expected.  Although the query below scans only 1 million rows, it runs for 6,7254 seconds.# User@Host: msandbox[msandbox] @ localhost [127.0.0.1] Id: 188 # Schema: Last_errno: 1160 Killed: 0 # Query_time: 67254.081455 Lock_time: 0.000006 Rows_sent: 1 Rows_examined: 1000000 Rows_affected: 0 Bytes_sent: 0 ... SELECT COUNT(*) FROM db1.sbtest1 WHERE id = 1 OR id = 2 OR id = 3 OR id = 4 OR id = 5 OR id = 6 OR id = 7 OR id = 8 OR id = 9 ... ...The CPU profile indicates that 99% of CPU time was spent in Item_cond_or::val_int() on MySQL 8.4.connection └ mysql_execute_command → Sql_cmd_dml::execute → Query_expression::ExecuteIteratorQuery └ AggregateIterator::Read 99.80% └ FilterIterator::Read 99.80% ├ Item_cond_or::val_int 98.96% │ └ Item::val_bool → Item_func_eq::val_int 91.27% │ └ Arg_comparator::compare_int_signed 58.46% │ ├ Item_field::val_int 16.18% │ └ (literal) val_int 10.04% └ IndexScanIterator::Read → row_search_mvcc 0.02% How MySQL evaluates OR conditions MySQL turns id = 1 OR id = 2 OR ... OR id = N into one Item_cond_or holding N Item_func_eq  elements.longlong Item_cond_or::val_int() { assert(fixed); List_iterator_fast<Item> li(list); Item *item; null_value = false; while ((item = li++)) { if (item->val_bool()) { null_value = false; return 1; } if (item->null_value) null_value = true; if (current_thd->is_error()) return error_int(); } return 0; }It checks each OR  element in the list until it finds a match. If a row matches term k, it costs k. If a row matches nothing, it costs N. The cost of the linear search is O(N). How MySQL evaluates IN conditions bool in_longlong::find_item(Item *item) { if (m_used_size == 0) return false; packed_longlong result; val_item(item, &result); if (item->null_value) return false; return std::binary_search(base.begin(), base.begin() + m_used_size, result, Cmp_longlong()); }The cost of the binary search is O(log N). The cost of OR conditions MySQL evaluates OR conditions in a nested loop. The outer loop walks the rows from the table, and the inner loop walks the OR list:for each row r: -- outer loop: R rows for each term o: -- inner loop: up to N terms if r == o: break -- exit inner loop; continue with next rowAssume that N is the number of values in the OR list R is the number of rows the filter evaluates T is the number of rows in the table A row that matches term k leaves the inner loop after k checks, which is about N/2 on average. A row that matches nothing runs all N. The OR list costs O(R × N). R depends on the query’s access type. How access type impacts the cost Range access: O(N²) When the optimizer chooses range access type, it reads only the matching keys from the index:mysql > explain for connection 11\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: sbtest1 partitions: NULL type: range possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 34753 filtered: 100.00 Extra: Using where; Using index 1 row in set (0.00 sec)The range has already done the filtering, but the server checks each row again, term by term. R grows with N, and each row checks about N/2 terms: R × N/2 ≈ N²/2. R is equal to N when all OR conditions in the table exist. In general, the cost is O(N²). Since the table size T does not appear in this cost, the increasing number of rows does not increase the execution time. The same query on tables of 100k, 500k, and 1M rows takes about the same time: Although the number of rows in the table grows 10 times from 100k to 1M, the execution time is still close. Range access reads only the N matching rows, so only N matters. However, R is not always equal to N. R is less than N when only part of the list exists in the table. R is greater than N when each value matches several rows. In this case, the cost is O(p × N²), where p is the percentage of rows matched from the table. Index scan: O(T × N) When the optimizer chooses index access type, it reads all records from the index:mysql > explain for connection 11\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: sbtest1 partitions: NULL type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 1000000 filtered: 100.00 Extra: Using where; Using index 1 row in set (0.00 sec)Now R = T. Every row in the table goes through the inner loop, and the T − N rows that match nothing run all N terms. The cost is O(T × N). T does not change as N grows, so the cost is linear in N. Linear does not mean cheaper, because T is much larger than N. At N = 40,000 on this table of about 1M rows, range access checks about N²/2 = 0.8 billion terms. The index scan checks about T × N = 40 billion, roughly 50 times more. Going from 34,753 OR conditions to 34,754 OR conditions pushed the ranges past the limit. The OR query went from 37 s to 2,907 s, which is 79 times longer.   In the index  access type, the table size T is part of this cost, and the test shows it. The same 40,000-value OR query with an index scan took 3,406 s on a 1M-row table, 1,752 s on a 500k-row table, and 265 s on a 100k-row table. Full table scan: O(T × N) This is similar to the index scan, which occurs when no suitable index is available for the query.mysql> explain for connection 9\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: sbtest1 partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1000000 filtered: 100.00 Extra: Using where 1 row in set (0.01 sec) Why does the optimizer switch from range to index Reason 1: the range_optimizer_max_mem_size  (8 MB by default) limit is exceeded.mysql > show warnings limit 1\G *************************** 1. row *************************** Level: Warning Code: 3170 Message: Memory capacity of 8388608 bytes for 'range_optimizer_max_mem_size' exceeded. Range optimization was not done for this query. 1 row in set (0.00 sec)In this case, the server raises warning 3170 and does not use range access. Please read more blogs about it Low Value in range_optimizer_max_mem_size Might Cause Full Table Scan Why Are Queries with Many IN Values More Expensive After Upgrading to MySQL 8.x? Reason 2: the cost model thinks the index scan is cheaper. The optimizer trace indicates it does not choose the range scan because of its cost."range_scan_alternatives": [ { "index": "PRIMARY", "ranges": [ "id = 1", ... "id = 9998", "id = 9999", "id = 10000" ], "index_dives_for_eq_ranges": false, "rowid_ordered": true, "using_mrr": false, "index_only": true, "in_memory": 0.000709723, "rows": 10000, "cost": 11006.7, "chosen": false, "cause": "cost" } The cost of IN conditions In general, the cost of evaluating the entire IN condition is O(R log N).  Similar to OR conditions, R depends on the query’s access type. After rewriting the query with IN, the execution time is below. Conclusion Since the time complexity of OR conditions is more expensive than that of IN, the query with OR conditions is slower than the query with IN conditions.   The post Why MySQL OR is slower than IN appeared first on Percona.

  • Building PlanetScale with Insights Agent
    We've always used PlanetScale to build PlanetScale. So when we built Insights Agent, our own databases were the first place we ran it to see what kind of gains we could get. Insights Agent identifies problematic queries or schema using Query Insights, finds the code responsible for them, and opens a pull request with a fix. Our Rails API is a large, busy app, which makes it a good testing ground.We know a thing or two about databases over here, so our databases are already quite well optimized. Even so, Insights Agent has been able to find a bunch of wins for us, continuing our drive towards the most efficient database possible.Here are three (of many) examples of how we tested and used Insights Agent in the weeks leading up to launch, and how we used these experiences to make the product better.A feature flag check we didn't needOur Rails API checks feature flags with Flipper. On top of Flipper, we have an override that can turn a flag off for specific records. Checking the flag requires reading it from the database. Our code checked the override first, every time:Each lookup took about a millisecond, which doesn't raise any significant slow-query concerns at first look. However, when this happens ~28 million times per day, it added up to more than nine hours of database CPU time. Insights Agent found it by the aggregate time instead of individual execution time. The override can only turn a flag off, so Insights Agent moved it after the Flipper check. def feature_enabled?(name) - if override_disables?(name) - false - else - Flipper.enabled?(name, actor) - end + Flipper.enabled?(name, actor) && !override_disables?(name) end If a flag is already off, there's nothing to override. The app behavior remains exactly the same as before, but now with far fewer lookups on these values:BeforeAfterCalls per day28.7 million8.3 millionDatabase time per day33,379 s9,625 sShare of all query time3.6%1.1%In this case, the traffic volume to the database did not change, but by careful inspection of both the database load and the codebase, the Insights Agent was able to significantly reduce load here.An index that made a query slowerNext, we asked Insights Agent to fix a slower query. This query looks up billing prices for an account, about 570,000 times a day (names simplified):select prices.* from prices where prices.owner_id = ? and prices.metric in (<elements>) and prices.owner_type = ? and prices.region_id is not null Insights showed this query reading about 122 rows for every returned row. This is often a tell that a query can be sped up. Insights Agent observed that the index MySQL used for this lookup had a range on region_id ahead of metric, and decided to add a new index with metric first. This might seem reasonable at first glance, but we merged it, and the query got slower:Before the indexAfter the indexAverage27.3 ms47.6 msp99129 ms222 msMySQL wasn't using the new index!Insights Agent was missing... somethingInsights Agent was working from a summary that left out a few important bits of context.First, almost none of these calls return a row, because most accounts have no prices to show. When the number of rows returned is zero (or near-zero), rows read per row returned looks terrible.Second, our Insights extension collapses IN lists when it normalizes SQL, which means <elements> was hiding 645 metric names, about 18 KB of SQL on every call. MySQL builds a lookup range for each of these values for every index that could use them. The new index gave it one more index to do that for.So we stopped sending the list of elements. The query now loads the few dozen regional prices an account has and filters them in Ruby:account.regional_prices.select do |price| price.region_id == region.id && BILLABLE_METRICS.include?(price.metric) end Old queryNew queryDatabase time per day16,212 s1,129 sAverage28.5 ms1.95 msp99137 ms8.0 msThat's about 14 times less database time. The new query also uses the index Insights Agent added, so we kept it.Teaching Insights Agent what we learnedWe have updated Insights Agent to look at more evidence to be better at these types of investigations. For these, it must look at rows read and returned per call and the size of the SQL sent, including how many values are collapsed. And we're extending our eval harness to cover cases like this one, including the cases where the best option is to leave the query as-is. Sometimes no change is better than adding an index.More indexing workYet another indexing adventure here. A job in our Rails API runs every minute to find schema snapshots that need to be reconciled (query simplified):select snapshot.id from snapshot join branch on branch.id = snapshot.branch_id where snapshot.ready = ? and snapshot.failed_at is null and snapshot.created_at > ? and branch.ready = ? order by snapshot.id asc limit ? The only index MySQL could use for this was one created on (ready, migration_applied_at). This index helps MySQL find unready snapshots. However, it does not help with the other filters because failed_at and created_at are not in it (as well as a few others which were removed). MySQL had to read every unready snapshot (ready = false) and check the additional columns on every row.There are a lot of unready snapshots in the database, but only a very small subset are both recent and not failed. MySQL had to find every match before it could sort by id and return the first few. Each call was reading 100s of thousands of rows to return, typically, less than 100 and was taking multiple seconds. The rows read / rows returned ratio on this query was really bad.Insights Agent suggested adding an index on (ready, failed_at, created_at):add_index :snapshot, [:ready, :failed_at, :created_at] With ready and failed_at now included, MySQL can jump straight to the right range of created_at instead of reading every unready snapshot. Another query that counts unready snapshots for alerting had a similar issue, and was given a fix by the agent. In this case, the only change needed to the app-side code was a database migration.Both p50 and p95 dropped from seconds per call to millisecond-scale.WrapupWe spent weeks testing, improving, and optimizing the experience with Insights Agent in the weeks leading up to launch. While doing so, we used it to automate the discovery of a bunch of places where we could improve our code, our database schema, and our queries. If you want a faster database (and we know you do) go give Insights Agent a try on your database, today.If you prefer doing all these kinds of optimizations in your own agent harness, we have some docs to get you up and running on this as well.