Web Operations at a startup, the first 30 days…

Hitting the ground running

I’d recently found myself working at a startup fulfilling their needs for Web Operations. I thought I’d document and give advice for anyone else in the same situation.

Most howtos give you advice for the first 100 days. By most conventional wisdom, you need 90 days to really figure out what’s going on in an organization. Most medium and large organizations let you do this at a relatively leisurely pace. Being at a startup where things are changing at a blistering place, it’s easy to get turned around.

Make Lists, get it out of your head

You’ll find new information daily. Information about things you’ll need to know. Information about things you don’t. You won’t really know what’s what at this point. What you will want to do is centralize it in a place like Evernote. I’m not saying use a notepad, but you’ll want to make it searchable because by the end of a month you’ll probably have forgotten much of what you wrote down at first.

I found that I had so much to think about it was actually keeping me up at night. Take a page from David Allen, and write this stuff down. Honestly, I made a list called “All The Things” since we didn’t even have a ticket tracking system at the time and wrote down, literally, all the things people wanted, or expected to get done that fell under my purview. You’ll get better sleep knowing you don’t have to hold anything in your head.

It won’t be enough to make a list of All The Things. You’ll want to prioritize them as well. You may want assistance from your manager on some of these. Try and figure out what you’ll do in 2 weeks, 4 weeks, etc. Don’t try and put too much on your plate, don’t change the world in a day. You’ll want to focus on quick wins when you can just to get some early recognition.


You’ve probably done this at your former position, don’t neglect it now. Figure how how your boss wants status weekly, and take this opportunity to do weekly one on ones with your directs. The longer you wait the less likely you’ll do it. Get good habits in place as soon as you can. It’s a new job. Make new habits!

Of course, also identify your peers and talk to them whenever possible. It’ll be easy in the future to have adverse relationships with them since their goals may not align with yours. However, they’ll also be most important allies in helping you and the company achieve goals.

Gather Subject Matter

What do you need to focus on. What’s lacking in the company. Do they need better documentation, monitoring, uptime, is there a ton of legacy stuff needing upgrading, are there too many requests being jammed through the hopper, all the above?

You’ll want to gather tasks and group them together so you can focus on like tasks at once. See the First Hundred Days Reference below, and don’t forget the Limoncelli test.

You’ll also want to document everything you find, especially if its lacking. Otherwise, read read read about everything that works there, and definitely use time in your one on ones to find any other bugs under the rug.

The First Hundred Days


The best uptime?

I found a box today in a squirreled away hosting provider that I’ve been paying the bill for mindlessly. I’ve found I can probably move the thing to s3 since the pages are essentially static. Amazingly enough, this may be one of highest uptime servers ever (maybe the most rooted server ever too but I don’t see any evidence yet). This host has amazing uptime. I think my company has experienced about 4 datacenter outages in the time this server has been up.

[root@localhost ~]# cat /etc/issue
Fedora Core release 3 (Heidelberg)
Kernel \r on an \m

[root@localhost ~]# uptime
 22:05:06 up 1595 days,  7:14,  2 users,  load average: 0.00, 0.00, 0.00

[root@localhost logs]# pwd

[root@localhost logs]# ls
archive                                 localhost_log.2006-10-19.txt
catalina.out                            localhost_log.2006-10-20.txt
host_count                              localhost_log.2006-10-21.txt
localhost_access_log.2006-05-12.txt.gz  localhost_log.2006-10-22.txt
localhost_access_log.2006-05-13.txt.gz  localhost_log.2006-10-23.txt

[root@localhost ROOT]# date
Wed Jul 11 22:36:37 PDT 2012

Pretty amazing stuff. Unfortunately it’s time to consolidate and put this guy to bed. Of course IP addresses and server names have been deliberately hidden to protect the innocent.

Replacing macbook pro feet

Now for something less business and profound and a little more practical.

Some folks have lamented about the feet on their Macbook Pro falling off. This works on a 15″ mid 2010 model. I’ve seen folks say that you can get these at the Apple store. Unfortunately, the only advice I’ve seen was to get a brand new bottom plate. This costs about $200. This is fine if you want a perfect bottom, but it’s really friggin spendy. Others have used felt bottoms, which doesn’t really have the right texture and will slip around.
My method involves scissors, electric tape, and a dime.
You can probably see where I’m going here. A dime is around the same size as the feet on a unibody macbook pro. What I did was cut smaller circles to fill in the hole where the foot was, so that it’s sufficiently raised. About 5 layers of electric tape circles will do. To cover it all, put the tape on dime, cut around it, and slap it on top. It’ll have about the same texture as the other feet, and will be fairly close to what was there. Enough to let your laptop be grippy enough till the next one you buy.

You really need a system if you’re going to make it work

Sometimes going through not just the day, but the week, I start really to sympathize with Leonard from Memento. If you haven’t seen the film I highly recommend it. As many of you know it’s about a guy who’s wife was killed, and he’s searching for the killer, but the catch is his short term memory is around 30 minutes max.

He creates a system to remember things, tattoos, notes, a polaroid camera and a sharpie.

“(For) just for day-to-day stuff, notes are really useful. Sammy Jenkis had the same problem but he really had no system. He wrote himself a ridiculous amount of notes but he’d get them all mixed up. You really do need a system if…you’re gonna make it work.”

For just managing yourself, it’s not hard. For a sysadmin, you’ve got Getting Things Done, Time Management for System Administrators. These are good systems, but I’ve always had a hell of a time making them work. This isn’t to say don’t read these books. If you’re looking at ideas for productivity, read them. Just don’t expect them to be the end all be all of task and project management. For me with weekly status reports I create, these methods don’t quite fit. Though the systems I use are heavily inspired from these systems. 

The systems tend to have days compartmentalized onto separate files. It becomes a pain to transfer tasks from one day to the next, to the next. Eventually it’s as if nothing gets done because I get very little done that I planned on that day. Quite demoralizing. Also since each day is separate, context from previous days needs to be transferred daily. That is the flow of the week is on each file, so I’m flipping between different notes for requests and things I did previous days.

Instead I have a list I just keep in Evernote that keeps track of the entire week. What do I want to do that week? What do I want my team to do that week? Then I have below the days of the week what I want to do, and what I did. Hey most of my status is one one page now, I can just format it. Taking the week in chunks, I’ve found I can get more things done. Realistically tasks will probably get done in a day or two, sometimes three. However a week as a nice even chunk of time to keep track of the stuff I’m doing. I have to report on the week anyhow, so it fits.

Of course you still need some sort of project management. For this I’ve been using Asana which so far is about the least fuss project management app I’ve been able to find. It’s free to boot. Again I’ve been able to track which projects are occurring, who’s doing them (even if its me). It doesn’t do anything fancy like gantt charts, but I’ve found I’ve been able to live without them. The hardest part is just following up on missed dates, and getting people to update it with their current projects.

That’s really what’s necessary for groups that are doing projects. They need to know who’s doing what, and who’s on point for which task. I’ve had a very hard time making project management files say MS Project working, because not everyone knows what’s going on realtime. You want to communicate to folks realtime with each update you make? You won’t get much done.

The last tool is simply for customer requests, Request Tracker. Though I’m careful this is regulated only to customer requests. It’s not a wish list, it’s not a task list, it’s certainly not used to track projects. You can probably make it do these things, but in my experience this is a bad idea, and not very easy. I try and make this very title apropos since its strong point is being a liaison where a documented conversation between customer and fulfiller can exist.

Monitoring and Uptime

Some fun notes about monitoring and uptime.

I’ll be giving some shout-outs to some products which have really helped me out. I’m not getting anything from them, (in fact my company is paying them money). However they’re making my life a bit easier.

So in world not so long ago, our monitoring situation was pure Nagios, and a physical pager. The pager was a bit like the hot potato in that it marked that you were on call. May as well have been an albatross for all it was worth. The service was cheap enough, but of course we had to come up with creative ways to create the on call schedule. Distribute it to folks that needed it, etc. Now there are probably cool ways to page with Nagios, and I’m sure people are using them, however we decided to outsource it to PagerDuty. It’s one of those companies that makes every sysadmin “Gah, why didn’t I think of that?!”.

PagerDuty takes care of the oncall schedule, rotations, being able to easily tweak the rotation because your on call guy wants to take vacation that week, or is otherwise unavailable, as well as the alerting you parts. It can call you, SMS you, e-mail you, all three at once. You get to pick. On top of that their rates are very reasonable. If you’re using a real pager, take a serious look at PagerDuty. I’ve got about 35 folks on it, multiple on-call rotations, and it’s working quite nicely.

Second shout out is for Keynote. You may be using Nagios (or Icinga) to monitor your uptime, and hey they have nifty performance metrics too! If you’re monitoring this stuff from the same datacenter that your website is on, you’re doing it wrong if you’re trying to figure out what your availability is.

Consider this (real) scenario. You’re tasked with providing marketing or sales, or whomever with you how awesome your site uptime is. Maybe it’s 100% and you don’t even do any outside monitoring, in which case you’re lucky or a liar. One day the power goes out in your datacenter, the whole place is dark for a bit, and of course it takes you a while to get your systems back up. Your monitoring system is blissfully unaware that there’s any site downtime because it’s down too. You go to pull up your reports and you still have 100% (or close to it) uptime.

The biggest issue is that Nagios is great for introspection. Making sure the gears in your machine are operating normally. It’s pretty crappy for monitoring what your customers on the internet are experiencing, and that’s what Keynote is good at. Other nice thing that Keynote does is dig through multiple clicks of a website with a simulated or real browser. Again emulating the customer experience as close as possible.

Hopefully this helps some folks out there. I know this took me longer than it should have to get right.

Using zen-like focus for minimizing distractions

For those of us in the world of tech, we tend to be busy folks these days. Some of us more than others, but at least in my work there is a lot going on all the time. The key is to know what deserves your attention, and what doesn’t. Every distraction is seemingly benign. A spam here, browsing a small site there, doing a little this and that. It adds up before you know it and pretty soon your day is over, and you’re wondering “What did I do all day? I know, I was just so *busy*”. Yes you were busy, but unfortunately you were busy much of the day doing the wrong things.
There have been several developments out there that call to attention this problem. Lifehacker has some tools to minimize distractions.  OS X Lion now has a full screen mode for its apps to remove create a distraction free environment to read or create. While tech tools are just one step in helping out, they’ll only get you most of the way there. So what to to do?
The tech tools are the easiest to implement. The hard part about minimizing distractions is that this isn’t a 5 minute, or even 1 hour activity. Instead, this is a continuous process. When, for example, you’re reading your e-mail, really decide if any of these are a waste of time. A lot of us tech guys get many of these and yes it’s easy to just hit the delete key, but if you’re doing this daily, or hourly, and this isn’t adding real benefit, you’re doing it wrong. Remember the sysadmin axiom that if you’re doing anything repetitive more than once, you should consider automating it.
The next step is the toughest. You’ll need to train yourself to focus, really hard, one one thing at at time. Make this your mantra. Unitasking is something you should be proud of. There have already been many studies that humans are horrible at multitasking. Multitasking simply doesn’t work, and if you have a boss that is convinced otherwise, you’ll need to disavow them of this notion. You need to focus on unitasking.
This means to focus on a single task till completion for as long as possible. If you can do this for 90 mins at a time, you’re likely well ahead of many.  You may even consider using a task timer to show yourself how well you’re unitasking. If you’re aware during this task that you’re finding yourself wandering towards other things, this is normal. Steer yourself back to the task at hand if the others can wait. In fact, this sort of unitasking is not unlike zen style meditation. Clearing your mind. Focusing on a single thing deliberately.
Another key to this style of unitasking is to schedule when you’ll be doing things to increase your mental discipline. Yes, you do need to take mental breaks during your long tasks. Again this can vary from person to person. Some folks can do 30 mins at a time, others much longer. Those mental breaks are a great opportunity to get up, stretch, focus your eyes elsewhere. Some folks will take a mental break by doing some social networking, and a little minesweeper, or tetris but this doesn’t work for a couple reasons.
  • You’re still engaging your brain, and doing nothing for your eyestrain, but now you’re doing something else.
  • You’re annoying your boss and peers by such behavior. They think you’re just goofing off, and admit it. You are.
Schedule when you will be doing these fun things however. Everyone wants to know what’s going on with their friends, and in the world. Nothing wrong with that, but just regulate it to certain times of the day. Likewise for the rest of your tasks, schedule them out so you know what you’ll be doing. The more structure you’re able to put here, the more chances for succeeding in what you sought to do that day you’ll have. I highly recommend Getting Things Done and Time Management for System Administrators to help with creating systems for task lists.
Don’t forget that you will still need to make time to go through your e-mail, build up your spam lists. Create tasks to reduce those distraction creators. With discipline, you’ll be able to further increase your productivity, and show yourself at the end of the day that you really were busy doing the right things.

Why BSD is dying, or How I learned to stop worrying and love Linux

I’ve been a fan of FreeBSD for most of my adult life. Ever since I first had to install it back at my first job in the bay area. It was the mid 90’s. FreeBSD was version 2.2.1.
Now before then I’d messed with the venerable Slackware, and a little of Redhat. Slackware was a nightmare to install back then (maybe it still is). You didn’t download anything because well, how far are you going to get with a 33.6kbps modem anyhow? I fell in love with FreeBSD. It was easy to install. Package dependancies were easy, ports just worked them out. It was stable as hell. The community wasn’t bad either.
Something happened between then and the 2000’s. Some important folks have since left the project, such as Jordan Hubbard and Matt Dillion. Likely for their own reasons. The project itself isn’t dead, they’re still doing a lot. They’re just focusing on the wrong things.
Section6 no longer runs on FreeBSD. It was much due to the fact that I’ve migrated from my own hosted server to EC2. AMI’s for FreeBSD are out there, but it’s not considered stable or widely supported. Ubuntu, and Redhat will just work. This is a reoccurring pattern with the project. FreeBSD is still stuck in the 90’s for a few reasons:
  • There’s no rapid deployment system like Debian’s fai, or Redhat’s kickstart.
  • You still have compile all your ports.
  • An upgrade requires you to recompile the entire OS, and they rarely go smoothly.
  • A port management add-on community has sprung up because they refuse to integrate a better native package management system.
  • It’s still lagging behind for cloud applications.
  • Desktop applications are quaint because everyone knows the multimedia support is lagging.
For a pragmatic sysadmin such as myself, it’s become increasingly hard to justify running it. Especially since my day job consists of all Linux distributions. It just takes so long to do anything. That is FreeBSD’s greatest sin. An operating system that forces you to waste so much time, between common tasks for hundreds of servers these days, compiles, installs, for such little benefit, isn’t doing anyone a service. The only thing FreeBSD has going for it these days is zfs and jails. Even then, Linux has xen, openvz, and vserver. Soon btrfs will be production worthy which has pretty much feature parity with zfs. Even then you still can get native zfs on Ubuntu and other Linux.
So I’ve gone with Ubuntu. It took about 2 weeks to migrate everything. I can save that for another post, but let’s just say that I could have made it easier had I used something like cfengine from the start. I’ll miss the ole daemon in favor of the currently mascot-less Ubuntu. The community certainly is just as good if not better. I’ll still keep the old tutorials around, but I won’t be writing anymore for FreeBSD. It’s obvious there’s no tutorials needed for Ubuntu. There’s quite a tutorial writing community out there.