If someone gets access they can delete your keys, or set up something that can intercept your keys in other ways.

The security of data at rest is just one piece of the puzzle. In many systems the access to the data is considered much more important than whether the data itself is encrypted in one particular scenario.