🚨Recovery_prefetch Causes Outages in Postgres
Recovery_prefetch turns Postgres into an outage machine
TL;DR
Postgres's recovery_prefetch feature, designed to improve performance, can cause significant outages. Disabling it can prevent replay lag and IOPS exhaustion, but it's a trade-off.
Recovery_prefetch, a feature in Postgres, was developed in February 2021 and shipped a year and a half later. It converts sequential I/O into 8k random reads, amplifying IOPS on replicas by up to 16x. This can easily exhaust available IOPS, causing replicas to lag and potentially leading to outages. Disabling recovery_prefetch eliminates these issues, but it's a trade-off between performance and stability. If you're running synchronous replication, you should be wary of recovery_prefetch.
Key Points
Recovery_prefetch converts sequential I/O into 8k random reads, amplifying IOPS by up to 16x.
Disabling recovery_prefetch eliminates replay lag and IOPS exhaustion issues.
The feature was committed without raising a fuss in February 2021.
A customer reported significant replay lag in March 2025 due to recovery_prefetch.
Disabling recovery_prefetch prevents the database from hanging due to synchronous replication.
Why It Matters
If you're running Postgres with synchronous replication, recovery_prefetch can cause significant outages. Disabling it prevents replay lag and IOPS exhaustion, but it's a trade-off between performance and stability. The issue is particularly relevant for databases with high write activity and limited IOPS.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,569 builders reading daily.