Testing junkie and optimistic computer bricker.
8 stories
·
2 followers

A Downtime Irony

31 Comments and 44 Shares

So many things can go wrong and often do, but I spend a good third of my time working on infrastructure, monitoring, and analytics so that they don’t.

Here’s what happened: At approximately 4:30pm PT feed fetching ceased. The feed fetchers were still working, which is why my monitors didn’t fire and alert anybody. But I have a second large Mongo database server used exclusively for collecting data about feeds being fetched. There are approximately 75 servers dedicated to feed fetching. These analytics look at average fetch times on a per task server basis. I use these analytics to ensure that my task servers are humming along, as they each use a ton of network, cpu, and memory.

This Mongo analytics servers works in a curious way. If you right-click on a feed and go to Statistics you’ll see the feed fetch history for a feed, stretching back a hundred fetches if the feed has had any issues in fetching. I keep these statistics on an analytics server separate from the regular Mongo server. I do this so that if the mongo analytics server goes down, everything will operate normally.

But the mongo server didn’t go down. It merely gave this error:

OperationError: Could not save document (Can't take a write lock while out of disk space)

Mongo continues serving read queries while not allowing write queries. I didn’t plan for that! And it gets worse. The way MongoDB stores data is that is just keeps growing, even as you delete data. NewsBlur only saves the last few fetches, but deleting old fetches doesn’t give you back any disk space. Every other database server I use has an autovacuum process that takes care of this maintenance work (PostgreSQL, Redis, Elasticsearch, but not MongoDB). It’s unfortunate that this is yet another instance of MongoDB being the cause of downtime, even though the fault lies with me.

The server that is meant to only be used to ensure things are working correctly was itself the culprit for feeds no longer being fetched. This is the ironic part.

NewsBlur’s developer during happier times wearing the 2013 NewsBlur t-shirt in Big Sur

Now comes the painful part. On Wednesday morning (yesterday) I packed my car and headed down to Big Sur to go backpack camping for the first time. I’ve car camped plenty of times, but I felt confident enough to pack my sleeping bag and tent into a big bag and head ten miles into the woods of coastal California.

I headed out, away from cellular service, at 4pm PT, half an hour before the analytics server ran out of disk space. And then returned nearly 24 hours later to a bevy of alarmed tweets, emails, direct messages, and a voicemail letting me know that things were haywire.

But the real problem is that I set a vacation reply on both my personal and work email accounts to say that I’d be out until September 3rd. Now, I hired a firm to watch the servers while I’m at Burning Man starting this Saturday. But I figured I could get away with leaving the servers for twenty four hours. And I neglected to tweet out that I’d be gone for a day, so theories cropped up that I was injured, dead, or worse, ignoring the service.

Brittany, NewsBlur’s developer’s girlfriend, can handle any situation, including driving a hysterical developer three hours back to San Francisco without breaking a sweat.

If you’re wondering, I think about NewsBlur first thing in the morning and last thing at night when I check Twitter for mentions. It’s my life and I would never just give up on it. I just got cocky after a year and a half of nearly uninterrupted service. NewsBlur requires next to no maintenance, apart from handling support requests and building new features (and occasionally fixing old ones). So I figured what harm could 24 hours of away time be? Boy was I wrong.

If you made it this far then you probably care about NewsBlur’s future. I want to not only assure you that I will be building better monitoring to ensure this never happens again, but to also offer anybody who feels that they are not getting their money’s worth a refund. Even if you are months away from payment, if you aren’t completely satisfied and think NewsBlur’s just about the best thing to happen to RSS since Brent Simmons released NetNewsWire back in 2004, then I want to give you your money back and let you keep your premium account until it expires.

I would like to also mention how much I appreciate the more light-hearted tweets that I read while on the frenetic three hour drive back to San Francisco from Big Sur. I do this for all of your happiness. If I did it for the money I’d probably find a way to juice the data so that I could at least afford to hire an employee. This is a labor of love and your payment goes directly into supporting it.

Big Sur is where a good many new ideas are thought.
Read the whole story
untriangle
3756 days ago
reply
Paris
Share this story
Delete
28 public comments
rafeco
3754 days ago
reply
Don't feel bad, databases gonna database.
laza
3755 days ago
reply
Don't feel horrible, this just showed us how valuble Newsblur is for our workflow or procrastination :) , and i'm so jealous that you are going to Burning Man, you should write a blog post about your experience there.
Belgrade, Serbia
samuel
3743 days ago
Writing that post right now. My work made the national news a few times. Look for the heartbeat lotuses.
emdot
3755 days ago
reply
I absolutely love NewsBlur. Thanks to Samuel for the update -- things like this make me even more happy to use the service. Thanks for your hard work.
San Luis Obispo, CA
satadru
3755 days ago
reply
I'm considering it a hallmark of the intensity of the academic program I'm in that I didn't much notice the Newsblur downtime.
New York, NY
jbouvier
3756 days ago
reply
Any developer knows downtime happens. The only thing you can do is your best to prepare, and when the shit hits the fan, be honest about the cause. This is how all developers & companies should treat these inevitable occurrences.
Chicago, IL
chengjih
3756 days ago
reply
Hah, I am so far behind on my feeds I didn't even notice the issue. 6000+ unread articles before the outage, 6000+ after.
llucax
3756 days ago
Same here, I don't have as many unread stuff, but this is just a RSS feed reader, come on! Let this guy have a day off!
loic
3756 days ago
reply
Murphy + mauvais timing + gestion de crise = super post
France
smadin
3756 days ago
reply
Well, here's what happened with Newsblur.
Boston
acdha
3756 days ago
reply
This is such a perfect sysadmin story. Kudos to Samuel for sharing the details.
Washington, DC
smilerz
3756 days ago
reply
Murphy strikes again.
Chicago or thereabouts
sirshannon
3756 days ago
reply
Exactly.
Cafeine
3756 days ago
reply
That combo of bad timing and technical glitch is just crazy. -_-;
Paris / France
Brstrk
3756 days ago
reply
Meh. No worries here. Murphy happens to everybody, man.
bronzehedwick
3756 days ago
reply
No intention of asking for a refund. :)
Tarrytown, NY
sredfern
3756 days ago
reply
Here here.
Sydney Australia
lkraav
3756 days ago
reply
What needs to happen in order for Sam to be able to hire an employee? How much would the yearly rate go up?
Tallinn, Estonia
3756 days ago
Thats a question I also want to raise. There definitely need some kind of business continuity for situations like that (or even worse one). I really appreciate the work Samuel is doing here and I love Newsblur, but I also see that its to heavily depends on one person.
samuel
3743 days ago
Going 24 hours without internet and not paying a server babysitter is pretty new. I often go without access for a few hours, but I almost always have Twitter and email on me and can post status messages. This was a one-time thing.
lkraav
3743 days ago
Well I think there comes a time in every genius' life where a delegation sidekick just leaves more energy for saving the world from bad feed readers.
jlvanderzwan
3756 days ago
reply
Gotta work on your bus factor, Sam! :)

http://en.wikipedia.org/wiki/Bus_factor
wmorrell
3756 days ago
reply
It is honest self-assessments like these, and the overall openness of NewsBlur, that makes it better than GReader for me. I remember a few downtimes there that were real downtimes, not just "it takes slightly longer to view feeds".
lasombra
3756 days ago
reply
Samuel, it sucks, but it's part of the job. I love NewsBlur and paid for a premium account after 1 hour of using it. I'm on the second run and all I have to say is "Great job man!"
UK
pablooo
3756 days ago
reply
$PWD
DrewCPU
3756 days ago
reply
The whole day, I thought "Poor Sam, he's gotta be going nuts!" I hope you enjoyed your much-deserved offline day.

Meanwhile, I was looking at all of my feeds manually and realized that I didn't need some of them and did some cleanup.
New Jersey
kazriko
3753 days ago
Hah. I was doing the same, but mostly moving sites that haven't updated in years to my "Dead Sites" folder. I keep them there just so I can see if something suddenly revives on me.
rewingau
3756 days ago
reply
The main thing I learnt from this downtime was just how much I value Newsblur. Hint - a lot!

And as for the downtime - the demon Murphy laughs at your camping trip...
Canberra, Australian Capital T
alannashaikh
3756 days ago
reply
This is really no big deal. Stuff happens.
rtreborb
3756 days ago
reply
Hard to be upset with you, Samuel. Thanks for all you do!
San Antonio, TX
tedder
3756 days ago
reply
it's all good. you have a great story to tell at a conference now.
Uranus
samuel
3756 days ago
reply
Ugg, I feel so horrible about the downtime, and so soon after feeling so wonderful.
Cambridge, Massachusetts
sredfern
3756 days ago
I'd rather you have a holiday and I deal with a couple of hours of downtime :)
brucealdridge
3756 days ago
you deserve a holiday ... and 75 feed servers?!?!
samuel
3756 days ago
That's only the feed fetchers. I also have a few dozen apps servers and a dozen different database servers. NewsBlur's a hungry beast!
larand
3756 days ago
Stuff happens. Thank you for all the work you've put into NewsBlur, and I have absolutely no intention of asking for a refund. Hell, I'd renew early if need be.
mp4328
3756 days ago
have an amazing Burning Man experience!! - don't worry about us while you're out there.
jqlive
3756 days ago
No worries man. Stuff happens. Enjoy Burning Man. Thanks for all the hard work.
rikishiama
3756 days ago
no worries here too. great honest post that makes me feel good I'm a year-plus paying user.
[deleted]
3756 days ago
Ditto to all these comments!
acdha
3756 days ago
Enjoy a break. The rest of us can use some practice dealing with not clicking refresh like a rat in a behavioural study…
kpjackson
3756 days ago
Let me add my note of appreciation for your honesty and dedication to this great product. Downtime happens and we can all learn lessons from our mistakes. It's when we don't learn that we should really feel horrible. Oh, and I plan to be a happily paying customer for years to come. Also, any plans for a 2014 NB Tee?
ScottInPDX
3756 days ago
Having been in similar situations in the past, I totally understand. You're doing good work here, and NewsBlur is the only reader I use. I'll be paying for it as long as you're willing to accept payment.
samuel
3756 days ago
I would love to get a 2014 NewsBlur t-shirt out there, but I haven't found a designer yet. Most want $1000+ for a design, and I'm telling them there's only about a hundred of these things that are going to sell. Happy to hear ideas!
[deleted]
3755 days ago
So sorry that it had to happen while you were away. Thanks for the status update - but mostly, thank you for the wonderful service you've built. It's invaluable to me (I feel like I owe you more than I pay.) Thanks!!!
rubin110
3756 days ago
reply
Thanks for rocking the thin blurry line between open source and able to financially support you and your work. In all honesty a random day of down time is a blessing with the number of feeds I've got going. Thanks again for all you've done.
San Francisco, CA, USA
kazriko
3756 days ago
reply
No chances for requesting a refund here. The service is too good, and has been too stable. This is a really unusual bit of downtime. Thanks for being on top of it for so long. I was still able to fetch and read the important feeds by instant-fetching then looking at each feed separately, so no real issue other than being time consuming.
Colorado Plateau
denubis
3756 days ago
I completely agree. Downtime happens, and developers deserve vacations. This is a wonderful service.
g_hoges
3756 days ago
+1 from me too. I love the service, and tolerate far more downtime from things like banking platforms. Continue to rock, Samuel
lasombra
3756 days ago
Right on man. Downtimes happen and Samuel has been rocking on the downtime arena for a very long time. It wasn't the end of the world. It sucks, but happens.
murrayhenson
3756 days ago
Seconded. Also, Samuel took the time to quickly comment on what went wrong and to really accept responsibility. You can't buy that but it's worth a hell of a lot.
JamesDiGioia
3756 days ago
And the thing about it is it's not strictly "downtime" - you could access the service, and with some minor hacky workarounds, continue to use it. Definitely not asking for a refund.
JimB
3754 days ago
I quite agree with you. Everyone is entitled to a break from the grind, and *sometimes* computers do unexpected things. The amazing thing is that you sorted out so quickly once you found out.
cmarshall
3753 days ago
Every system has issues occasionally - the important thing is how you handle them. The two keys are communication and getting it fixed. You done good.

xkcd Phone

8 Comments and 21 Shares
Presented in partnership with Qualcomm, Craigslist, Whirlpool, Hostess, LifeStyles, and the US Chamber of Commerce. Manufactured on equipment which also processes peanuts. Price includes 2-year Knicks contract. Phone may extinguish nearby birthday candles. If phone ships with Siri, return immediately; do not speak to her and ignore any instructions she gives. Do not remove lead casing. Phone may attract/trap insects; this is normal. Volume adjustable (requires root). If you experience sudden tingling, nausea, or vomiting, perform a factory reset immediately. Do not submerge in water; phone will drown. Exterior may be frictionless. Prolonged use can cause mood swings, short-term memory loss, and seizures. Avert eyes while replacing battery. Under certain circumstances, wireless transmitter may control God.
Read the whole story
untriangle
3868 days ago
reply
Paris
Share this story
Delete
8 public comments
petrilli
3867 days ago
reply
I want my phone to scream when I drop it.
Arlington, VA
jepler
3867 days ago
reply
I so want the "alternate speed of light" feature.
Earth, Sol system, Western spiral arm
amijangos
3868 days ago
reply
Love the side facing camara!
Columbus, Indiana
rraszews
3868 days ago
reply
Warning: Do not taunt the Happy Fun xkcd Phone
Columbia, MD
Brstrk
3868 days ago
reply
Yeah, but how long does the battery last?
JayM
3868 days ago
reply
.
Atlanta, GA
[deleted]
3868 days ago
??
SmartManne
3868 days ago
reply
Oh God...
rohitt
3868 days ago
Won't help. "Under certain circumstances, wireless transmitter may control God"
srsly
3868 days ago
reply
Alt Text

"Presented in partnership with Qualcomm, Craigslist, Whirlpool, Hostess, LifeStyles, and the US Chamber of Commerce. Manufactured on equipment which also processes peanuts. Price includes 2-year Knicks contract. Phone may extinguish nearby birthday candles. If phone ships with Siri, return immediately; do not speak to her and ignore any instructions she gives. Do not remove lead casing. Phone may attract/trap insects; this is normal. Volume adjustable (requires root). If you experience sudden tingling, nausea, or vomiting, perform a factory reset immediately. Do not submerge in water; phone will drown. Exterior may be frictionless. Prolonged use can cause mood swings, short-term memory loss, and seizures. Avert eyes while replacing battery. Under certain circumstances, wireless transmitter may control God."
Atlanta, Georgia
republicson
3868 days ago
Bwaa haaa!
ridingsloth
3868 days ago
Why isn't there already an app that makes my phone scream when I drop it?
[deleted]
3868 days ago
@ridingsloth... i'd buy that app!

tar

6 Comments and 9 Shares
I don't know what's worse--the fact that after 15 years of using tar I still can't keep the flags straight, or that after 15 years of technological advancement I'm still mucking with tar flags that were 15 years old when I started.
Read the whole story
untriangle
4323 days ago
reply
Paris
Share this story
Delete
6 public comments
samuel
4320 days ago
reply
I learned the handy tar cvf to make tarballs.
Cambridge, Massachusetts
aamermoquim
4322 days ago
reply
faced this yesterday.
Alternate Universe
brico
4322 days ago
reply
Lolololo
Brooklyn, NY
adamgurri
4323 days ago
reply
Unix jokes lulz
New York, NY
greggrossmeier
4323 days ago
reply
I hate to admit it, but I exemplified this just today.
Ojai, CA, US
4310 days ago
It is muscle memory, the only part I remember is e(x)tract and (c)reate. And when I create, I always list the order of tarball and files incorrectly. ^_^
sredfern
4323 days ago
reply
tar xfvz file
Sydney Australia
michaelglass
4323 days ago
HONESTLY!
euser
4323 days ago
Be careful, older versions of tar expect the 'f' to be in front of the file argument, so better make it -xzvf
skorgu
4323 days ago
'tar t' would be even safer.
aripollak
4322 days ago
No need to specify z, it will automatically detect the compression.
euser
4317 days ago
That's a very late GNU feature @aripollak, so don't rely on it when they say "Unix" here (and GNU's Not Unix) ;)

When your house is burning down, you should brush your teeth

3 Comments and 9 Shares
When your house is burning down, you should brush your teeth

20 years ago today my house burned downed, so I wrote a comic about it

View
Read the whole story
untriangle
4347 days ago
reply
Paris
Share this story
Delete
3 public comments
grammargirl
4346 days ago
reply
Matt Inman: kind of a dick. This story's pretty good, though.
Brooklyn, NY
Door
4346 days ago
reply
This is perhaps my favourite thing The Oatmeal has done. Sad and funny and very real.
Vienna, VA, USA
kazriko
4347 days ago
reply
...
Colorado Plateau
trekkie
4347 days ago
yep.

Mark Shuttleworth: Ubuntu in 2013

1 Share

This is a time of year to ponder what matters most and choose what we’ll focus on in the year to come. Each of us has our own priorities and perspective, so your goals may be very different to mine. Nevertheless, for everyone in the Ubuntu project, here’s what I’ll be working towards in the coming year, and why.

First, what matters most?

It matters that we not exclude people from our audience. From the artist making scenes for the next blockbuster, to the person who needs a safe way to surf the web once a day, it’s important to me, and to the wider Ubuntu community, the people be able to derive some benefit from our efforts. Some of that benefit might be oblique – when someone prefers XFCE to Unity, they are still benefiting from enormous efforts by hundreds of people to make the core Ubuntu platform, as well as the Xubuntu team’s unique flourish. Even in the rare case where the gift is received ungraciously, the joy is in the giving, and it matters that our efforts paid dividends for others.

In this sense, it matters most that we bring the benefits of free software to an audience which would not previously have had the confidence to be different. If you’ve been arguing over software licenses for the best part of 15 years then you would probably be fine with whatever came before Ubuntu. And perhaps the thing you really need is the ability to share your insights and experience with all the people in your life who wouldn’t previously have been able to relate to the things you care about. So we have that interest in common.

It matters that we make a platform which can be USED by anybody. That’s why we’ve invested so much into research and thinking about how people use their software, what kinds of tools they need handy access to, and what the future looks like. We know that there are plenty of smart people who’s needs are well served by what existed in the past. We continue to maintain older versions of Ubuntu so that they can enjoy those tools on a stable platform. But we want to shape the future, which means exploring territory that is unfamiliar, uncertain and easy to criticise. And in this regard, we know, scientifically, that Ubuntu with Unity is better than anything else out there. That’s not to diminish the works of others, or the opinions of those that prefer something else, it’s to celebrate that the world of free software now has a face that will be friendly to anybody you care to recommend it.

It also matters that we be relevant for the kinds of computing that people want to do every day.

That’s why Unity in 2013 will be all about mobile – bringing Ubuntu to phones and tablets. Shaping Unity to provide the things we’ve learned are most important across all form factors, beautifully. Broadening the Ubuntu community to include mobile developers who need new tools and frameworks to create mobile software. Defining new form factors that enable new kinds of work and play altogether. Bringing clearly into focus the driving forces that have shaped our new desktop into one facet of a bigger gem.

It’s also why we’ll push deeper into the cloud, making it even easier, faster and cost effective to scale out modern infrastructure on the cloud of your choice, or create clouds for your own consumption and commerce. Whether you’re building out a big data cluster or a super-scaled storage solution, you’ll get it done faster on Ubuntu than any other platform, thanks to the amazing work of our cloud community. Whatever your UI of choice, having the same core tools and libraries from your phone to your desktop to your server and your cloud instances makes life infinitely easier. Consider it a gift from all of us at Ubuntu.

There will always be things that we differ on between ourselves, and those who want to define themselves by their differences to us on particular points. We can’t help them every time, or convince them of our integrity when it doesn’t suit their world view. What we can do is step back and look at that backdrop: the biggest community in free software, totally global, diverse in their needs and interests, but united in a desire to make it possible for anybody to get a high quality computing experience that is first class in every sense. Wow. Thank you. That’s why I’ll devote most of my time and energy to bringing that vision to fruition. Here’s to a great 2013.

Read the whole story
untriangle
4360 days ago
reply
Paris
Share this story
Delete

Click and Drag

8 Comments and 10 Shares
Click and drag.
Read the whole story
untriangle
4458 days ago
reply
Paris
Share this story
Delete
8 public comments
llucax
4457 days ago
reply
One of those epic and historic xkcd!
Berlin
grammargirl
4458 days ago
reply
Wow.
Brooklyn, NY
simonft
4458 days ago
reply
This may help:
http://xkcd-map.rent-a-geek.de/
NYC
tedder
4458 days ago
perfect. better than writing it myself or using the github archive.
glenn
4458 days ago
reply
Wow, this is amazing! It just keeps going and going and going...
Waterloo, Canada
afita
4458 days ago
reply
xkcd: always surpising.
Cluj-Napoca, România
huckncatch
4458 days ago
reply
OMG… it never ends!
timlikescake
4458 days ago
reply
Perhaps the best thing about this is that you know there are stacks of jokes which you've missed, but you know you'll never find them because there's no way to zoom.
ChrisDL
4458 days ago
This is also the worst thing about this. (to be clear, I am a fan of xkcd, not saying it's bad just frustrating to click around knowing I won't find everything OCD)
timlikescake
4458 days ago
Absoutely. I had a quick dive through the source code to see if there was an easy way to see the whole thing at once.
timlikescake
4458 days ago
I also misclicked after about 15 minutes and ended up losing my place =/
slivergun
4458 days ago
Personally i like it, you need to FIND the jokes.
ChrisDL
4458 days ago
For sure @slivergun,BUT you can never be absolutely sure you have found ALL of them!!! o_O;;
krpt
4458 days ago
some1 released offline version, https://github.com/danielribeiro/xkcd_grab
ChrisDL
4458 days ago
@krpt Praise you good sir!
krpt
4458 days ago
@ChrisDL you're welcome mate ^_^
Next Page of Stories