pg_resetxlog: When hope depends on luck

PostgreSQL has proven to be one of the most reliable Open Source databases around. In the past we have dealt with customers who have achieved superior uptimes and who are really happy with PostgreSQL. But even if PostgreSQL does its job day after day after day – there are some components, which can just fail and cause trouble. In short: Hardware may crash, filesystem can fail, and so on.

In 99% of all cases there is a way around total disaster. Either you have a replica, which protected you from downtime or you simply have a reasonable good backup to help out in case of disaster. But what if there is really no option left anymore? You got to make this database work again – no matter what (even if it is just a partial or an unreliable restore). Maybe pg_resetxlog is your last helping hand in this case.

Killing an instance

As you might know the purpose of the transaction log (= xlog or WAL) is to restore order and consistency in case of a crash. If the xlog is broken, you might face disaster. This is exactly what we want 😉

So, to cause some disaster I sent a “kill -9” to a database system under load to stop it. In case of a restart this would lead to a recovery process and PostgreSQL would fix things on startup by replaying the transaction log. Let us disturb the replay process a little …

[hs@jacqueline pg_xlog]$ dd if=/dev/zero of=00000001000000050000007C bs=1024 count=16384

16384+0 records in

16384+0 records out

16777216 bytes (17 MB) copied, 0.0280045 s, 599 MB/s

[hs@jacqueline pg_xlog]$ dd if=/dev/zero of=00000001000000050000007D bs=1024 count=16384

16384+0 records in

16384+0 records out

16777216 bytes (17 MB) copied, 0.0304789 s, 550 MB/s

[hs@jacqueline pg_xlog]$ dd if=/dev/zero of=00000001000000050000007F bs=1024 count=16384

16384+0 records in

16384+0 records out

16777216 bytes (17 MB) copied, 0.028223 s, 594 MB/s

What I have just done is really nasty. I have overwritten some transaction log files found in the pg_xlog directory with zeros. This should instantly make sure that the recovery process ends up in hell. Let us check if my attempts to really kill things has succeeded:

[hs@jacqueline test_db]$ pg_ctl -D /data start

server starting

LOG:  database system was shut down at 2014-08-08 12:17:00 CEST

LOG:  invalid magic number 0000 in log segment 00000001000000050000007B, offset 0

LOG:  invalid primary checkpoint record

LOG:  invalid magic number 0000 in log segment 00000001000000050000007B, offset 0

LOG:  invalid secondary checkpoint record

PANIC:  could not locate a valid checkpoint record

LOG:  startup process (PID 18807) was terminated by signal 6: Aborted

LOG:  aborting startup due to startup process failure

Wow, this is exactly what I have intended to do. Nothing works anymore so it is time to fix things and demonstrate the power of pg_resetxlog.

pg_resetxlog at work

pg_resetxlog is a C program shipped with PostgreSQL. So, there is no need to install it on top of PostgreSQL. It is simply there and ready for action in case it is needed. Actually it is very powerful as the following help page shows:

[hs@jacqueline ~]$ pg_resetxlog --help

pg_resetxlog resets the PostgreSQL transaction log.

 

Usage:

  pg_resetxlog [OPTION]... DATADIR

 

Options:

  -e XIDEPOCH      set next transaction ID epoch

  -f               force update to be done

  -l XLOGFILE      force minimum WAL starting location for new transaction log

  -m MXID,MXID     set next and oldest multitransaction ID

  -n               no update, just show what would be done (for testing)

  -o OID           set next OID

  -O OFFSET        set next multitransaction offset

  -V, --version    output version information, then exit

  -x XID           set next transaction ID

  -?, --help       show this help, then exit

It is possible to nicely configure pg_resetxlog to make it do precisely what is needed (maybe by just setting xlog back a little). However, in this example we don’t care too much – we just want to reset the xlog completely:

[hs@jacqueline test_db]$ pg_resetxlog /data

Transaction log reset

One line is enough to reset the xlog of PostgreSQL.

Let us see if this has worked:

[hs@jacqueline test_db]$ pg_ctl -D . start

server starting

LOG:  database system was shut down at 2014-08-08 12:18:51 CEST

LOG:  database system is ready to accept connections

LOG:  autovacuum launcher started

Voila, the database was able to start up again. The autovacuum daemon has been launched nicely and the system is able to accept connection. A simple test reveals that all databases seem to be around:

[hs@jacqueline test_db]$ psql -l

                              List of databases

   Name    | Owner | Encoding |   Collate   |    Ctype    | Access privileges

-----------+-------+----------+-------------+-------------+-------------------

 postgres  | hs    | UTF8     | en_US.UTF-8 | en_US.UTF-8 |

 template0 | hs    | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/hs            +

           |       |          |             |             | hs=CTc/hs

 template1 | hs    | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/hs            +

           |       |          |             |             | hs=CTc/hs

 test      | hs    | UTF8     | en_US.UTF-8 | en_US.UTF-8 |

(4 rows)

So, where is the problem?

Be cautious

pg_resetxlog is really the last line of defense. If you have to call it you should expect some data to be lost. Maybe you will face slightly broken tables or corrupted indexes. You should not really trust this database anymore – however, you will have the chance to extract some data, restore at least something.

Under any circumstances: If you are forced to use pg_resetxlog I would really advise to not continue with the database instance in doubt, take a backup, PROPERLY check the data and start over with a new database instance.

Hans-Juergen Schoenig
Hans-Jürgen Schönig has 15 years of experience with PostgreSQL. He is consultant and CEO of the company „Cybertec Schönig & Schönig GmbH“ (www.cybertec.at, www.postgresql-support.de), which has served countless customers around the globe.