I am not a system administrator. However, I am a software engineer, and responsible for administrating systems. As technical director, it is also my responsibility to create policies. Knowing that systems can be compromised, I know a policy needs to be in place for protection. Not allowing all engineers to run constantly as the root or super user on a Linux box is a good place to start. What happens though when I do not heed the warning of my own policy?
The answer to that is simple. At midnight I end up submitting an urgent support ticket to our hosting provider to have our dedicated box reconfigured. By reconfigured, I mean, a clean reinstall of the OS. If you are not familiar with Linux systems, the command rm -r /*, when run as a super user, will wipe an entire server clean. I typed the command inadvertently while attempting to delete the contents of a directory while it was the working directory, without deleting the directory itself.
I know the command. It has crossed my path several times on IT forums and Web sites like The Daily WTF. Frantic developers typically want to know how to undo the tragedy, hoping that deep in the recesses of their Linux distribution there must be a system restore feature. There is no undo button for such a blunder, and I already knew it. So the question becomes, what went wrong? If I am writing the policies, and I expect others to follow them, why the self-destruct?
Step away
The timeliest excuse is an inability to step away. I was trying to install an application between an evening dessert and bedtime. I grew frustrated after having several difficulties with the procedure, and started pouring over a knowledgebase and forums. I was tired, irritable, and losing patience. Add into the mix that this was not a crucial task. Without question it could have waited until the next morning. The end result is a slip of the fingers on the keyboard, and a sickening realization.
As hard as it may be to step away from an incomplete task in the midst of a struggle, do it. No matter your talent or your skill set, it could save you a sleepless night. Come back to the job refreshed, and with a clear head. If you manage other engineers, then there will be occasions where you must ask them to leave the work at work. Deadlines do not always permit this flexibility, but if they always prohibit your engineers from stepping away, then you have other fish to fry.
Never ever
If anything can go wrong, it will. This adage, known as Murphy’s Law, has become commonplace. However, the inevitability implied is no longer given much weight in Western society. There is not a programmer today who has not at one point in his career sidestepped traditional wisdom by taking a shortcut. This thinking, especially in conjunction with the occasional necessity to step away from the keyboard, is a dangerous combination.
I distinctly recall the thought that I should add an SSH user only for managing the domain on our server. It would prevent the need to perform simple confined tasks as the root user, eliminating significant risk. I delayed, knowing that I would create a policy for others at a future date. Because only other programmers would need such a policy. Never ever would I need this failsafe. Needless to say, that SSH user exists today.
The options
There is one final point with regard to making policies, which I will call the Law of Diminished Responsibility. It basically states that relinquishing responsibility is often best when a third-party has already taken on the same responsibility. We do it every day. It can also be called passing the buck. Later I discovered that our hosting company is a trusted partner with the organization that developed the application I was trying to install. They would install it for free, and accept the responsibility.
In the future as we work with this hosting company, and I attempt to free up resources and time, it would make sense that I ask them to perform the installation. Being open to options, even if it means relinquishing authority and responsibility, can eliminate the need to even create a policy. I will still need to install some applications, but weighing all the options first is an absolute requirement. It does not always reduce risk, but it can reduce responsibility in taking risks.
A nod
This post would not be complete without a nod to Nexcess.net, my company’s hosting provider. I was leery that they had no 24/7 phone support, but they do have a ticket system monitored 24/7 by staff. I received a response within the hour, and by morning the OS was reinstalled. Thanks guys.
The problem here is that there exists no easy mechanism to disallow such.
rm -rf / should simply be configurable.
I know i would ALWAYS turn off a question dialog emerging “do you really want to remove ‘/’? [y/n]“
@markus
there reason why a message isn’t displayed is b/c the person used the -f option, which tells rm to delete without prompting (forced delete).
For more information, see the man page.
http://unixhelp.ed.ac.uk/CGI/man-cgi?rm
There already exists a way to disallow rm. Use a ****ing alias.
As for logging in as root… what the hell? sudo(8) exists for a reason.
How would an alias help? He would be used to delete a directory (and in that way, his system) by using the alias.
Same for sudo.
[...] http://blog.reindel.com/2008/05/31/a-lesson-in-policymaking/ [...]
I dunno man. Policy is OK but there is no such thing as preventative policy. Look at it from the perspective from someone who will be responsible for laying their own fingers on the keyboard every day.
When a mistake happens, and it will, a preventative policy becomes useless. It’s not even a deterrent because when it comes down to getting it done, the policy goes out the window (and consequently creates the myths of “legend”).
A “restorative” orientation is a stronger approach for policy making. Assume lasseiz-faire, assume Murphy’s, assume something that your policy will never cover. Create a policy that motivates people that success is getting rest and taking a break. Create a policy that empowers people to push back on those who might unknowingly pressure dangerous situations. Create a policy that backs up your servers.
Hey Shaun,
Those are some great points! One thing I did not discuss was the need to have a backup plan in place. This touches on the restorative orientation you mentioned. Something will go wrong — the question is, what will you do when it happens?