How to manage distributed services?
What do you use to keep applications runing when you have dozens of them running on a dozen different servers?
The case I am specifically asking about is when you have a dozen linux servers, and you have equally as many (and probably more) services that run on one or more of those servers. Those applications are a mix of home grown apps, cots products, and FOSS software.
So how do you keep them running, know what runs where, and start/stop them all without ending up with a nightmare?
Do you stick with init scripts (and roll your own)?
Do you use daemontools?
A homemade restart menu?
Let heartbeat manage it for you?
What works for you / what doesn’t?