I switched from a big custom Ansible deployment to NixOS.

The system includes 8 managed machines, multiple VPNs and a custom certificate authority.

Downsides:

• rethinking of how to manage Certificates and VPN configs outside of Nix
• getting secrets to work took a bit until I found agenix
• deployments can take a while with deploy-rs

Still, I can only tell you how much more at ease I feel with the NixOS based system. Its just much easier to refactor, not having to take care of legacy cleanup and polluting the machines over time.

Once you wrap your head around it all more complex system architectures start to become manageable/maintainable.

IaC

You still need sth like Terraform on the side for your actual infrastructure provisioning.

Solutions to bridge this with the Nix ecosystem are evolving in the nix-community repos on Github, but I found it easier to manage that separately for the time being.

All in all I would recommend NixOS based systems for the heavy lifters in your setup. If you want to deploy a fleet of machines you are entering new territory. Exciting, but maybe too much of a time commitment for some.

You could give https://nixos-mailserver.readthedocs.io/en/latest/ a try. If you are not familiar with NixOS it will be quite the learning curve, but can ultimately pay off if you host many things and want to gain transparency and peace of mind :)