Why

While working with a client, Unsupservised.com, we frequently provision Postgres databases on either Amazon RDS or using Crunchy Data's Postgres Operator.

When you deploy a sufficient number of large Postgres instances within a Kubernetes environment, you may eventually encounter a situation where one of the Postgres processes dies with an exit code 9.

This is due to the Linux OOM (Out of Memory) killer terminating the process to prevent it from causing the entire operating system to crash.

Postgres documentation even acknowledges this phenomenon in their docs

Although this setting will not prevent the OOM killer from being invoked altogether, it will lower the chances significantly and will therefore lead to more robust system behavior. This is done by selecting strict overcommit mode via sysctl. A privalged pod running busy box can actually set this from a sidecar or init container. I discovered this from the Opensearch Helm Chart

sysctl -w vm.overcommit_memory=2

After making this change, we still encountered process termination but gained additional diagnostic data.

The current Postgres settings for huge pages using the following command:

psql -c "SELECT name, setting FROM pg_settings where name ~ 'huge'"
               name               | setting 
----------------------------------+---------
 huge_page_size                   | 0
 huge_pages                       | off
 shared_memory_size_in_huge_pages | 80

The output shows that huge pages are turned off, and the page size is defaulting to the OS, which is typically 2MB (or occasionally 1GB). Given the current page size and other memory settings, Postgres requires 80 huge pages.

To configure huge pages, you can use the following command:

sysctl -w vm.nr_hugepages=80

It's important to note that hugePages are an allocatable resource, and therefore, you must set huge pages before the Kubernetes Kubelet service starts.

Enabling huge pages involves a slight detour and can be accomplished through various methods. The most straightforward approach I found is by modifying the autoscaling launch template, which includes a user data script that you can configure.

This user data script configures Kubelet with the necessary information to join the cluster. By adding the sysctl command to enable huge pages above this line in the script, you can allocate huge pages to containers running on that node.

Make sure to set this new launch template as the default and then manually initiate the rollout of this new configuration.

Neither of this changes fixed our issues but are documented here for futre refrence.