If you're running geth on a stock Ubuntu install, you might be surprised that the system allocates swap memory when regular memory is available. To prevent this from happening you need to change the "swapiness" that the kernel uses to decide which kind of memory to allocate.

Follow the instructions in this answer, then restart geth

Concretely, do this

sudo swapoff --all
sudo sysctl vm.swappiness=0 
sudo swapon --all

Then start geth. You'll need to do this on every reboot.