Moved to @cynici@fosstodon.org

South Africa
Joined April 2019
Include

Only show posts containing:

Exclude

Hide posts containing:

Time range
-
Minimum likes
Postgres has two ways to estimate costs and row counts for postgres_fdw. When a query joins a foreign table, the local planner needs to know what that remote data looks like to pick a smart plan. postgres_fdw can get those numbers from two places. Option 1: Ask the remote server at planning time ``` ALTER SERVER remote OPTIONS (ADD use_remote_estimate 'true'); -- Or ALTER FOREIGN TABLE orders OPTIONS (ADD use_remote_estimate 'true'); ``` With this enabled, postgres_fdw sends remote EXPLAIN queries over the wire during the planning phase and adopts the remote planner's estimates. Because those estimates leverage the remote server's live statistics and indexes, the local planner discovers what the remote database can do efficiently. For instance, if you join 50 local customer IDs to a remote orders table with 1,000,000 rows and an index on customer_id, use_remote_estimate lets Postgres choose a parameterized nested loop. The remote query becomes: ``` SELECT id, total FROM orders WHERE customer_id = $1 ``` Instead of pulling 1,000,000 rows across the network into a local hash join, it executes 50 fast index lookups on the remote side. The tradeoff is planning overhead. It pays a network round-trip for a remote EXPLAIN (often 2–3 round trips per foreign table per query). Option 2: Pre-sample and store statistics locally ``` ANALYZE orders; ``` Running ANALYZE on a foreign table samples the remote table over the network and saves the column statistics locally in pg_statistic. Future query planning requires zero remote round-trips. The gotchas: • Autovacuum ignores foreign tables. It will never automatically analyze them, so put an explicit ANALYZE on a schedule (e.g., using pg_cron). • Local stats describe rows, not remote indexes. Local statistics tell the planner how many rows match a filter, but not that an index exists remotely to fetch them cheaply. For indexed parameterized lookups, use use_remote_estimate. For broad scans and analytical queries, local statistics are usually enough.
2
3
31
951
This is so true.
114
2,433
83
38,561
379,596
52
785
56
3,844
49,210
cheewai retweeted
You realize anger was sometimes just pain asking to be understood.
202
10
1,000
17,550
cheewai retweeted
— Marcus Aurelius
2
243
4
1,289
24,281
cheewai retweeted
Error 404: Humor Found😉
@expensiveaccideSep 29
4
42
8
719
70,583
cheewai retweeted
Where are all the clowns?
10
26
5
199
11,525
cheewai retweeted
I worked for this company in 2011. Their game was to be the only marketplace in town and use oligopoly power to maximize margin by charging obscenely and pushing creator payouts to $0. Today, obliterated by AI. The NYSE is delisting them. Karma does catch up, occasionally.
146
344
58
7,592
558,133
cheewai retweeted
HR: "The employee survey is 100% private." Manager the next day
377
9,377
321
106,831
2,220,946
cheewai retweeted
I noticed something
247
3,995
260
15,891
282,652
Postgres has the concept of infinity. When dealing with numbers, dates, or timestamps, it can be a clean, effective tool for schema design. In our post on the Ambiguity of NULL in Postgres, we explained how calculations or comparisons with NULL create a contagion, which causes the result of the expression to always yield NULL. While NULL represents an unknown or missing value, infinity represents a known, unbounded value that is larger than all finite numbers or dates (and -infinity for values smaller than all others). Postgres range types (daterange, tsrange) already handle open-ended bounds natively. But if you are using separate columns for unbounded start/end dates or values, consider using infinity instead of NULL. The "NULL Contagion" Problem The following is what we mean by "contagion". When using the greater than operator to compare a date with NULL, the value is NULL: ```sql SELECT DATE '2024-06-15' > NULL AS gt_null; -- (null) ``` Because a WHERE clause only keeps rows where the condition evaluates to TRUE, any row containing a NULL comparison gets dropped. The standard workaround is padding every query with `OR ending_date IS NULL`. If your schema uses 'infinity' to represent an open-ended date, you eliminate the need for those query workarounds. Using Infinity in Practice Here is a price history table that stores the open-ended row as `'infinity'::date`. The greater-than comparison returns true against it, because infinity is larger than every finite date: ```sql CREATE TABLE prices ( product_id int NOT NULL, price numeric(10,2) NOT NULL, valid_from date NOT NULL, valid_to date NOT NULL ); INSERT INTO prices VALUES (1, 9.99, DATE '2024-01-01', DATE '2024-04-01'), (1, 11.99, DATE '2024-04-01', DATE '2024-07-01'), (1, 14.99, DATE '2024-07-01', 'infinity'::date); SELECT price FROM prices WHERE product_id = 1 AND valid_from <= DATE '2099-12-31' AND valid_to > DATE '2099-12-31'; -- 14.99 ``` Key Behaviors to Keep in Mind • Sorting: infinity sorts logically as greater than any finite value (and -infinity as smaller). NULL will still sort higher than infinity unless you use NULLS last. • Indexing & Performance: Standard B-Tree indexes index 'infinity' natively. Queries filtering on ranges (e.g., valid_to > CURRENT_DATE) can utilize clean single-index scans without forcing complex OR clauses or bitmap index scans to capture NULL values. • MIN / MAX Aggregates: max() encountering 'infinity' returns 'infinity'. min() will evaluate finite values normally unless '-infinity' is present. • Arithmetic: Adding or subtracting days from 'infinity' still yields 'infinity'. Functions like date_trunc() preserve infinity. When using a function against infinity, ensure it behaves as expected. • Equality & Constraints: Unlike NULL, infinity is equal to infinity ('infinity'::date = 'infinity'::date is TRUE). This allows UNIQUE constraints and indexes to evaluate open-ended rows predictably. By setting NOT NULL, but allowing infinity on a column, you remove the headache double checking every query for a "OR ... IS NULL". An asside about Toy Story: While writing this, I realized "To Infinity and Beyond" was a essential to the irrational confidence of the character of Buzz Lightyear. There is no beyond infinity, but he doesn't care, he's going to pursue it anyhow.
5
4
32
1,391
😂😂😂
76
334
25
3,365
134,951
cheewai retweeted
New interactive mini-book: Go concurrency distilled It gives a brief overview of many concurrency topics in Go, along with examples you can run in the browser. There's also a PDF version. The book is AI-free. antonz.org/go-concurrency-di…
12
85
6
657
35,763
cheewai retweeted
127
11,866
499
40,118
954,570
O velho do bairro me disse: "Não se esqueça que a bússola foi inventada antes do relógio, porque a direção é mais importante que o tempo."
276
34,642
522
276,662
2,771,071
cheewai retweeted
6
143
1
3,481
119,345
cheewai retweeted
One of the favorite topics in SQL interviews is inventory management. Here is a good example: 𝗧𝗵𝗲 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻: Imagine you are helping the Inventory department to identify high or low stock levels at specific locations. 𝗧𝗼 𝘀𝗼𝗹𝘃𝗲 𝘁𝗵𝗲 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻, 𝘆𝗼𝘂 𝗻𝗲𝗲𝗱 𝘁𝗼: • Establish a threshold to define high/low (we will use 10 units for this example). • Calculate the stock level of each product • Calculate the average stock level across all stores. It sounds simple, but think again: You can't use a GROUP BY because you need the details for each product per store. 𝗧𝗵𝗲 𝘀𝗶𝗺𝗽𝗹𝗲𝘀𝘁 𝘄𝗮𝘆 𝘁𝗼 𝘀𝗼𝗹𝘃𝗲 𝘁𝗵𝗶𝘀 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝗶𝘀 𝘂𝘀𝗶𝗻𝗴 𝗣𝗔𝗥𝗧𝗜𝗧𝗜𝗢𝗡 𝗕𝗬. The PARTITION BY clause in SQL divides the result set into partitions or "windows" based on one or more columns. Each partition is treated separately for calculations performed by a window function. Unlike GROUP BY, it does not aggregate the data into a single row per group, meaning you can still see all the original rows in the output. 𝗪𝗵𝗮𝘁 𝗵𝗮𝗽𝗽𝗲𝗻𝗲𝗱 𝗶𝗻 𝘁𝗵𝗶𝘀 𝗾𝘂𝗲𝗿𝘆: • The query calculates the average stock level for each product across all stores using the PARTITION BY clause. This clause groups the data by ProductId to ensure the average is calculated per product. • These average stock levels are stored temporarily using a Common Table Expression (CTE). • The query checks each store's stock level against the average within its product group. • The final result shows each store's stock level, the average stock level for that product (calculated using PARTITION BY), and whether the stock is 'High', 'Low', or 'Normal'. PARTITION BY is like a GROUP BY, but you keep the details. Keep learning!
4
15
1
92
4,400
cheewai retweeted
New: A map of the most important skills in AI Engineering.
553
4,123
483
24,507
6,957,449
cheewai retweeted
Zak van der Merwe has spent 14 years building control planes at AWS. His blog is a candid look at what happens when your bookkeeping layer has to keep up with the scale of the cloud, and why some of the most interesting problems in distributed systems often hide in the most "boring" parts. allthingsdistributed.com/202…
8
45
4
303
28,014