Thoughts about autonomous transactions

When reading Gianni Ciolli’s wiki page about autonomous transactions this week I spent some time to think about things in more detail.

The idea of an autonomous transaction is actually quite simple:

START TRANSACTION;

some_sql_action;


START AUTONONOUS TRANSACTION;

INSERT INTO log_table VALUES (some_data);

COMMIT AUTONOMOUS TRANSACTION;


some_more_sql_action;

ROLLBACK;

The main advantage here is actually that you can commit the log entry without having to commit the parent transaction. This makes the log entry visible if the parent transaction fails badly. Reporting errors to a table is ways more robust in this case. Technically the idea floating around is to push transactions to a stack doing some push / pop operations as autonomous transactions come and go.

BUT: What should happen in this case?

START TRANSACTION;
SELECT some_data FROM some_table WHERE id = 1 FOR UPDATE;

START AUTONONOUS TRANSACTION;
SELECT some_data FROM some_table WHERE id = 1 FOR UPDATE; -- > problem
COMMIT AUTONOMOUS TRANSACTION;

some_more_sql_action;

ROLLBACK;

If the child transaction is really supposed to be fully independent of its parent this should actually never be possible. The parent transaction locks up a row which is then used by the autonomous transaction. This leaves us with two choices: Either the child transaction is NOT fully independent or you allow users to “kill” themselves by locking up the connection.

When being out on the road doing consulting it is sometimes impressing to figure out how little people actually know about how joins work. Especially outer joins seem to be a mystery for most people.

Here is an example which might bring some enlightenment:

test=# CREATE TABLE a (id int4);
CREATE TABLE
test=# CREATE TABLE b (id int4);
CREATE TABLE
test=# INSERT INTO a VALUES (1), (2), (3);
INSERT 0 3
test=# INSERT INTO b VALUES (2), (3), (4);
INSERT 0 3

We just insert a couple of rows and see how things work out.

Let us start with a simple LEFT JOIN:

test=# SELECT * FROM a LEFT JOIN b ON (a.id = b.id);
 id | id 
----+----
  1 |   
  2 |  2
  3 |  3
(3 rows)

The goal here is to get all rows from the left hand side and see which rows match on the right side. The “missing” value is filled with a NULL token.

One of the key misunderstandings is this magical ON-clause:

test=# SELECT * FROM a LEFT JOIN b ON (a.id = b.id AND b.id = 2);
 id | id 
----+----
  1 |   
  2 |  2
  3 |   
(3 rows)

To most people this result is surprising. It is important to notice that the filter is only applied to the right side. So, we get one more NULL value. This is pretty easy to see and comprehend if the usecase is that simple – but, in case of an outer-join orgy it is not that trivial anymore.

The example I have just shown is not what people usually expect. Here is an alternative:

test=# SELECT * FROM a LEFT JOIN b ON (a.id = b.id) WHERE b.id = 2;
 id | id 
----+----
  2 |  2
(1 row)

The WHERE-clause will filter on both columns at the same time. This will actually turn the LEFT JOIN into a useless operation because the WHERE-clause will filter stuff away anyway. NOTE: This is a mistake which can commonly observed; most people don’t get ON and WHERE right when it comes to outer joins.

FULL OUTER joins

FULL OUTER joins are the next bug miracle:

test=# SELECT * FROM a FULL JOIN b ON (a.id = b.id);
 id | id 
----+----
  1 |   
  2 |  2
  3 |  3
    |  4
(4 rows)

We take all rows from the left hand side, take the matches on the right hand side and add NULL values in case no matches are found. So far this is no surprise …

The problem usually starts when we try to make the ON clause a little bit more fancy:

test=# SELECT * FROM a FULL JOIN b ON (a.id = b.id AND b.id = 2);
 id | id 
----+----
  1 |   
  2 |  2
  3 |   
    |  4
    |  3
(5 rows)

Most people would not expect “3” to exist at all on the right hand side. As you can see it is moved to the NULL-part of the join. This is not too intuitive for most people. However, this is perfectly expected behavior.

If we filter in the WHERE clause we will again turn the outer join into a useless operation:

test=# SELECT * FROM a FULL JOIN b ON (a.id = b.id) WHERE b.id = 2;
 id | id 
----+----
  2 |  2
(1 row)

Maybe this will bring some enlightenment to some users out there.

test=# CREATE TABLE t_test AS SELECT * FROM generate_series(1, 10000) AS id;
SELECT 10000

First we create a small demo with some simple data.
Now we do our fancy cleanup. For the sake of security we do it inside a transaction:

test=# BEGIN;
BEGIN

How about analyzing data while it is being deleted? Here is how it works …

test=# WITH x AS ( DELETE FROM t_test RETURNING * ) 
    SELECT min(id), max(id), avg(id), count(*) FROM x;
 min |  max  |          avg          | count 
-----+-------+-----------------------+-------
 1 | 10000 | 5000.5000000000000000 | 10000
(1 row)

We put the DELETE inside a so called WITH-clause. WITH will give us a “virtual table” called “x” which contains the data returned by DELETE. Note, we used DELETE in combination with a  “RETURNING *” … so we essentially tell DELETE to give us the data we are about to kick. As we might want to do it for 20 mio rows or so it is definitely too much for a night-time reading exercise. So, we turn this data coming back from the DELETE into something we can then nicely analyze inside our SELECT part. Given this analysis it is a lot easier to judge if our DELETE has really touched the data we wanted it to touch. It greatly reduces the odds of breaking something late at night.

If we feel safe, we can nicely commit …

test=# COMMIT;
COMMIT

Sometimes people have to delete a lot of data. Common table expressions (CTEs) make this process a lot more reliable and robust.

Again, this is not sane design and it was never intended — nobody knew beforehand for how long this can be sustained. It just ended up a little scary .

Some 5 years ago we fired up some 8.1 database servers which have successfully served XX bio transactions since then. Running a large PostgreSQL database successfully on UNIX over a number of years is actually nothing special. So, why care? Well, this thing is special — and trust me, it was never intended to be …

Imagine you start designing a simple database system for a small company (maybe 10 people). Things should be scalable and simply work for the given purpose. You might want to use some partitioning and maybe do a little trickery with indexing. So far so good … nothing special.

But, sometimes things just don’t work out as expected. In this special case it means: End users started to behave somehow totally differently then expected — and: there is A LOT more growth then ever anticipated. So, given the totally unexpected way people use the database, given the seriously higher growth in both users and amount of data the number of tables / partitions in the PostgreSQL systems went up and up and up and up ….

This kind of growth is especially scary when you take into account that in case of high growth it is usually not too easy to make a serious change in architecture within a small period of time (there is never enough time). The business side is usually occupied with sales, important extensions required by the customers and so on. Given a (meanwhile) complex application it is not too easy to simply introduce some PL/Proxy layer to scale out more easily without investing a lot of time and (hardly available manpower). So, you are actually forced to “tweak” things a little to make sure that you can actually buy some time for a change which can reflect the new needs …

And months go by and you keep tweaking and months go by and at some point you end up with something like this:

db=# \x
Expanded display is on.

db=# SELECT count(*) FROM pg_tables;
-[ RECORD 1 ]--
count | 2574107

I want to point out again that this was never intentional … nobody EVER planned to do put over 2 mio tables in the very same PostgreSQL database on the very same box. This happened definitely WAYS beyond my control — again; it was never intentional and NEVER planned this way. But still, it is impressing to see how well PostgreSQL behaves under really disgusting workloads.

Files, files, files …

Let us think for a moment how PostgreSQL stores tables inside a database instance — they simply go into the same directory unless you use tablespaces:

postgres@db1:/var/lib/postgresql/8.1/main/base/16385$ ls | wc –l
8104310

How can the operating system handle that? Well, first of all you have to adjust some kernel parameters to make this work. Secondly you got to make sure that you filesystem uses a tree down to the file level — otherwise a “seq scan” on the file list will kill you during lookup. Fortunately this was properly set when the system was set up.

Again, this is not sane design and it was never intended — nobody knew beforehand for how long this can be sustained. It just ended up a little scary .

What really impresses me here is that this thing works like a charm. We did not have a single database crash in 5 years. Interestingly the fileystem is playing along well for years as well — no filesystem corruption for 5 years; it simply works like a charm while serving around 20 mio transactions a week. The load on the system is actually fairly mixed — there is a large number of small transactions but also a fair list of real killers (bulk load, creating / removing hundreds or tables at a time and so on).

Why not consolidate into larger tables?

Well, given the large number of tables people would actually think that using larger tables would make more sense. Given the load this is actually not quite true … recall, using a seriously partitioned structure you basically get a “free index pointer” to a chunk of data without actually having to index the data itself. If you want to read a million rows at once it is a lot cheaper to look up the right table containing and read it sequentially than to actually fetch 1 mio rows from an insane amount of data by traversing some insanely large index. A seq scan on the right 1 mio rows is a lot faster than to actually go through a (largely) uncached, giant index.

Tribute to PostgreSQL 8.1

It is seriously impressing how far you can squeeze even an old release such as 8.1 when things get really hard to manage. Given the vast number of improvements which went into PostgreSQL since 8.1 it is fun too imagine how much better and easier to handle a new release might be. Still, this kind of application is definitely a killer … and PostgreSQL can really show its potential.

test=# BEGIN;
BEGIN

… some action …

test=# SAVEPOINT a1;
SAVEPOINT

… some action …

test=# SAVEPOINT a2;
SAVEPOINT

… some action …

test=# SAVEPOINT a3;
SAVEPOINT

If you nest thousands and thousands of savepoints inside one transaction without ever releasing a single savepoint you will figure out that after some time the performance of your transaction is about to decline rapidly.

The reason is fairly simple. Here is a brief output of some profiling session:

Flat profile:

Each sample counts as 0.01 seconds.

 %   cumulative   self              self     total
time   seconds   seconds    calls   s/call   s/call  name
35.20      0.69     0.69 11442642     0.00     0.00  LWLockAcquire
28.06      1.24     0.55 11442642     0.00     0.00  LWLockRelease
20.41      1.64     0.40 11433942     0.00     0.00  SimpleLruReadPage_ReadOnly
 5.61      1.75     0.11 11443596     0.00     0.00  TransactionIdPrecedes
 4.08      1.83     0.08 11429226     0.00     0.00  SubTransGetParent
 4.08      1.91     0.08     4716     0.00     0.00  SubTransGetTopmostTransaction
 0.51      1.92     0.01     9728     0.00     0.00  slot_deform_tuple
 

Every additional savepoints adds some small additional effort needed inside the database to keep track of the transaction status and so on. so, if your transaction has accumulated thousands of unreleased savepoints you will see a lot of of heavy activity inside the backend which is simply related to locking. Nothing bad is going to happend and PostgreSQL will definitely survive 500 mio savepoints inside the same transaction — however, you might not be happy with the performance after all.

How to get around this issue:

test=# RELEASE SAVEPOINT a1;
RELEASE

once a savepoint is not needed anymore it can simply be released. this will resolve the extensive CPU consumption shown in the list above.

So, after all … thank you all for having so much faith in us and that you think we can do millions of savepoints inside a transaction . yes, we can — but, you should not necessarily do it.

It is interesting to see how many lines of code are sometimes needed to solve a problem. Long programs seem to be an especially common fetish in the Java world.

How would you approach a problem like that in PostgreSQL?

Here is a simplified version:

test=# CREATE TABLE t_room (id serial, name text, action text);
NOTICE:  CREATE TABLE will create implicit sequence "t_room_id_seq" for serial column "t_room.id"
CREATE TABLE



We create a table with an ID, some name and an action.

Now we feed some data:

test=# INSERT INTO t_room (name, action) VALUES ('hans', 'IN');
INSERT 0 1
test=# INSERT INTO t_room (name, action) VALUES ('paul', 'IN');
INSERT 0 1
test=# INSERT INTO t_room (name, action) VALUES ('james', 'IN');
INSERT 0 1
test=# INSERT INTO t_room (name, action) VALUES ('paul', 'OUT');
INSERT 0 1
test=# INSERT INTO t_room (name, action) VALUES ('john', 'IN');
INSERT 0 1
test=# INSERT INTO t_room (name, action) VALUES ('hans', 'OUT');
INSERT 0 1

This is already enough to make the fun start:

test=# SELECT *,
       sum(CASE WHEN action = 'IN' THEN 1 ELSE -1 END)
               OVER ( ORDER BY id) AS field FROM t_room;

 id | name  | action | field
----+-------+--------+-------
  1 | hans  | IN     |     1
  2 | paul  | IN     |     2
  3 | james | IN     |     3
  4 | paul  | OUT    |     2
  5 | john  | IN     |     3
  6 | hans  | OUT    |     2
(6 rows)

Basically we have defined 2 actions: Somebody can enter the room or he can leave it. Order is provided by the id we have used in the example. So, how can we translate this to a counter? First of all we have to define that we want to increment the counter by 1 if somebody goes in and decrement it by 1 if somebody leaves. now we can do a running total.

This can be achieved with a simple windowing function.

Sometimes problems which make people code for days can be solved in one line.

The challenge is actually quite simple: “We have to trace all changes to data and structure”. Well, the first thing does not seem like a huge problem but how about changes to data structures? Well, somebody would argue that …

SET log_statement TO mod;

… will do the job. This will display all DDLs and commands doing changes to data. However, consider …

test=# SET log_statement TO none;
SET

This is a perfectly valid SQL command leaving us with a problem when it comes to tracking and auditing. We cannot tell for certain who has modified what because logging of DDLs can be turned off by user interaction.

It seems some additional and reliable way to prohibit that is needed — alternatively some additional tracing / auditing mode might help as well. It does not seem to be too attractive to have permissions of GUC variables because it might add an extra level of complexity nobody might want.

What somehow surprised me was the fact the Microsoft gave some talks on some Open Source components provided by them — mostly in the area of open government data and cloud-compliance. It is amusing to see how bad the sentiment towards those people actually was … but,  after all — my PostgreSQL talk had at least twice as many listeners than the “Open Data” talk provided by Microsoft before .

This was a fairly interesting weekend. This years “Linuxwochen” made a stop in Vienna and people enjoyed 3 days of interesting presentations and talks.

What somehow surprised me was the fact the Microsoft gave some talks on some Open Source components provided by them — mostly in the area of open government data and cloud-compliance. It is amusing to see how bad the sentiment towards those people actually was … but,  after all — my PostgreSQL talk had at least twice as many listeners than the “Open Data” talk provided by Microsoft before .

I gave a talk called “PostgreSQL roadshow” (video will be available soon) which showed some new technologies in the PostgreSQL area such as KNN, PgOpenCL, writable CTEs and so on. It seems people are highly interested in some stuff which goes seriously beyond “key / value”.

Last week I have successfully answered the “can you nest subselects”-question for the 52342424285675^th time so i decided to move “yes we can” a little further and turn it into “yes, we REALLY can” — here is the proof:

 
WITH x AS ( SELECT row_number() 
    OVER (PARTITION BY (SELECT * 
        FROM ( SELECT 1) AS e) ORDER BY 1) 
            FROM ( SELECT 1 d) AS y)
    SELECT ( SELECT * FROM ( SELECT * FROM x) AS x) AS output
        FROM ( SELECT 1 AS z) AS x
        WHERE z >= ( SELECT 1)
            OR z IN ( (SELECT 1), (SELECT 1))
        ORDER BY ( SELECT 1)
        LIMIT ( SELECT 1 FROM ( SELECT * FROM x) AS a);

[hs@lola ~]$ psql test < /tmp/file.sql 
 output 
--------
      1
(1 row)

I think we have fully proven now that PostgreSQL can REALLY nest subselects. If this is not enough evidende – step up now or keep silent forever