Welcome Back to Head-Fi.org. I'm Sorry About the Extended Outage
Nov 25, 2007 at 10:18 AM Thread Starter Post #1 of 353
[size=large]Welcome Back to Head-Fi.org. I'm Sorry About the Extended Outage.

As I'm sure many of you probably realized, Head-Fi.org has been down for two weeks now, after a catastrophic storage appliance failure that affected both Head-Fi.org and some of my work-related products/services. Long story short, the nature of the storage appliance failure rendered most of our backups corrupt, so we spent a tremendous amount of time cobbling together the site as best we could for a re-launch. Yes, it has taken this long, and we've been working with very little sleep since the moment it went down, divided up into teams, with Head-Fi'er heyimallan coordinating almost the entirety of the recovery effort. Frankly, we couldn't have done it without him. Thank you, heyimallan.

Our first priority was recovering the posts/threads/registrations--this data received the most care, attention and priority, as it constitutes the bulk of the content we created together as a community. Luckily, heyimallan was able to recover posts/threads/registration data all the way up to the moment of the big crash, which, again, for us, was the most important task.

Unfortunately, there were data casualties:
  1. We lost a chunk of the private message data. For example, I don't have any of my private messages after some time in January 2007 within vBulletin.
  2. Thread subscription data was lost.
  3. The photo.head-fi.org data is still in the process of trying to be recovered. If we can not recover it, we will likely discontinue photo hosting, as there are now a number of large, stable dedicated free photo hosting services, and this would be the second time we lost photo data (the first time happened during a software upgrade, during which photos became disassociated with profiles).
  4. Post attachment data was lost. I don't believe we are going to be able to recover this data, but we are still looking into it. I think this will be most painful to the DIY forums, where attachments were most commonly used.
  5. Personal profile equipment field data was lost. On the bright side, most of us could have probably updated that information anyway (I know mine wasn't very current).
  6. I'm still not sure how current the Contributing Membership data is within vBulletin, as I haven't had a chance to check that yet. If it's not current, I can update this information through PayPal records.
  7. Blog data was lost. However, I was able to recover almost all of the publicly viewable blog posts from Microsoft Live cached pages and Google cached pages. I have saved these as MS Word files, and will e-mail the content to the respective posters (like LFF, Sasaki, _j_, etc.), so that they can re-post them in their blogs.
  8. All of the URLs have changed--yes all--so web search engine results won't initially point to the right posts and threads. Additionally, you'll find that some of the hand-coded links in the templates still have to be fixed. I'll get to these.
I'm sure there's more that I'm missing right now, and I'll update as soon as I can.

You'll notice some broken images, changed icons, etc. We'll get to fixing these with time. There's so much more to do back here to tune things up to get us as close to normal as we can get us.

I want to thank my entire team, especially joe, jack, heyimallan, and Michael (who's not a Head-Fi'er) for all their efforts, both on our work side, and on the Head-Fi side. Again, however, I want to give the most credit for this Head-Fi recovery to heyimallan, who brought tremendous expertise and leadership (and countless hours of sacrificed sleep) to the recovery effort--again, both for my work-related services and Head-Fi.org.

All I can do at this point, everyone, is apologize to all of you (and to the Sponsors and other Members of the Trade who depend to some degree or another on the community we have here) about the extended outage (by far our worst and most catastrophic outage and loss in over six years), and tell you that we have taken several measures to do all we can to eliminate the possibility of this ever happening again.

I don't think I'll be posting much more tonight. I'm going to continue working to adjust and tidy things up around here, and then I'm going to get the heck out of this datacenter, go home, and get some much needed sleep.

I'm sorry about those things we have lost (mentioned above), but am thrilled that we have all our posts/threads/registrations.

I missed you all, and will be glad to see you all again.

Nov 25, 2007 at 10:34 AM Post #3 of 353


Head-Fi's Most Prolific Poster
Jan 13, 2004
Jude, its great to be back!
I am sure you guys have worked really hard the last 2 weeks, and have done a great job recovering almost all data.

Nov 25, 2007 at 10:35 AM Post #4 of 353


Headphoneus Supremus
May 25, 2006
Thanks a LOT Jude for all the hard work.

It's great to be back!
Nov 25, 2007 at 10:37 AM Post #5 of 353


100+ Head-Fier
Apr 27, 2006
Jude, thank you so much for the hard work you've put in recently.

I'm very grateful for Head-fi, as I'm sure everyone else is. It's really something special, and your hard work and perseverance is nothing short of admirable.

Once again, thank you so much!
Nov 25, 2007 at 10:39 AM Post #6 of 353


New Head-Fier
Sep 23, 2007
It is great to see the site is up and running again. It must have been a nightmare for for the whole crew. Thank you for all the work you have put in.
Head-fi is a great community and I am sure many of us will appreciate it even more now.

Nov 25, 2007 at 10:43 AM Post #11 of 353


Headphoneus Supremus
Jun 29, 2007
man i missed everyone, i didnt know what to do with myself...


good job guys, i know the type of ordeal you have gone through and we are lucky to have any semblance of original data back!!

great job
Nov 25, 2007 at 10:52 AM Post #14 of 353

Sweet Spot

1000+ Head-Fier
Feb 16, 2003
Jude, I felt so bad for you during this whole time, thinking of what it must be like to have something which means so much to you, cause you so much grief and pain... I'm really happy that you guys sorted it all out and really thank you for putting in so much time and effort in order that this great community continue to thrive.

You have nothing to be sorry to us about. We all know that these things happen, and that no one is to blame. Great job man, well done. And I hope everybody who was involved in getting things back on track reads my post as an equal thank you to them, as well.

Nov 25, 2007 at 10:53 AM Post #15 of 353


100+ Head-Fier
Jan 23, 2003
Thank you guys! And welcome back! I missed this site and all on it.

From my view, no apologies needed. I bet it will be veeery nice to sleep.


Users who are viewing this thread