- Joined
- Jun 20, 2001
- Posts
- 11,051
- Likes
- 7,031
[size=large]Welcome Back to Head-Fi.org. I'm Sorry About the Extended Outage.
[/size]
As I'm sure many of you probably realized, Head-Fi.org has been down for two weeks now, after a catastrophic storage appliance failure that affected both Head-Fi.org and some of my work-related products/services. Long story short, the nature of the storage appliance failure rendered most of our backups corrupt, so we spent a tremendous amount of time cobbling together the site as best we could for a re-launch. Yes, it has taken this long, and we've been working with very little sleep since the moment it went down, divided up into teams, with Head-Fi'er heyimallan coordinating almost the entirety of the recovery effort. Frankly, we couldn't have done it without him. Thank you, heyimallan.
Our first priority was recovering the posts/threads/registrations--this data received the most care, attention and priority, as it constitutes the bulk of the content we created together as a community. Luckily, heyimallan was able to recover posts/threads/registration data all the way up to the moment of the big crash, which, again, for us, was the most important task.
Unfortunately, there were data casualties:
You'll notice some broken images, changed icons, etc. We'll get to fixing these with time. There's so much more to do back here to tune things up to get us as close to normal as we can get us.
I want to thank my entire team, especially joe, jack, heyimallan, and Michael (who's not a Head-Fi'er) for all their efforts, both on our work side, and on the Head-Fi side. Again, however, I want to give the most credit for this Head-Fi recovery to heyimallan, who brought tremendous expertise and leadership (and countless hours of sacrificed sleep) to the recovery effort--again, both for my work-related services and Head-Fi.org.
All I can do at this point, everyone, is apologize to all of you (and to the Sponsors and other Members of the Trade who depend to some degree or another on the community we have here) about the extended outage (by far our worst and most catastrophic outage and loss in over six years), and tell you that we have taken several measures to do all we can to eliminate the possibility of this ever happening again.
I don't think I'll be posting much more tonight. I'm going to continue working to adjust and tidy things up around here, and then I'm going to get the heck out of this datacenter, go home, and get some much needed sleep.
I'm sorry about those things we have lost (mentioned above), but am thrilled that we have all our posts/threads/registrations.
I missed you all, and will be glad to see you all again.
Sincerely,
Jude
[/size]
As I'm sure many of you probably realized, Head-Fi.org has been down for two weeks now, after a catastrophic storage appliance failure that affected both Head-Fi.org and some of my work-related products/services. Long story short, the nature of the storage appliance failure rendered most of our backups corrupt, so we spent a tremendous amount of time cobbling together the site as best we could for a re-launch. Yes, it has taken this long, and we've been working with very little sleep since the moment it went down, divided up into teams, with Head-Fi'er heyimallan coordinating almost the entirety of the recovery effort. Frankly, we couldn't have done it without him. Thank you, heyimallan.
Our first priority was recovering the posts/threads/registrations--this data received the most care, attention and priority, as it constitutes the bulk of the content we created together as a community. Luckily, heyimallan was able to recover posts/threads/registration data all the way up to the moment of the big crash, which, again, for us, was the most important task.
Unfortunately, there were data casualties:
- We lost a chunk of the private message data. For example, I don't have any of my private messages after some time in January 2007 within vBulletin.
- Thread subscription data was lost.
- The photo.head-fi.org data is still in the process of trying to be recovered. If we can not recover it, we will likely discontinue photo hosting, as there are now a number of large, stable dedicated free photo hosting services, and this would be the second time we lost photo data (the first time happened during a software upgrade, during which photos became disassociated with profiles).
- Post attachment data was lost. I don't believe we are going to be able to recover this data, but we are still looking into it. I think this will be most painful to the DIY forums, where attachments were most commonly used.
- Personal profile equipment field data was lost. On the bright side, most of us could have probably updated that information anyway (I know mine wasn't very current).
- I'm still not sure how current the Contributing Membership data is within vBulletin, as I haven't had a chance to check that yet. If it's not current, I can update this information through PayPal records.
- Blog data was lost. However, I was able to recover almost all of the publicly viewable blog posts from Microsoft Live cached pages and Google cached pages. I have saved these as MS Word files, and will e-mail the content to the respective posters (like LFF, Sasaki, _j_, etc.), so that they can re-post them in their blogs.
- All of the URLs have changed--yes all--so web search engine results won't initially point to the right posts and threads. Additionally, you'll find that some of the hand-coded links in the templates still have to be fixed. I'll get to these.
You'll notice some broken images, changed icons, etc. We'll get to fixing these with time. There's so much more to do back here to tune things up to get us as close to normal as we can get us.
I want to thank my entire team, especially joe, jack, heyimallan, and Michael (who's not a Head-Fi'er) for all their efforts, both on our work side, and on the Head-Fi side. Again, however, I want to give the most credit for this Head-Fi recovery to heyimallan, who brought tremendous expertise and leadership (and countless hours of sacrificed sleep) to the recovery effort--again, both for my work-related services and Head-Fi.org.
All I can do at this point, everyone, is apologize to all of you (and to the Sponsors and other Members of the Trade who depend to some degree or another on the community we have here) about the extended outage (by far our worst and most catastrophic outage and loss in over six years), and tell you that we have taken several measures to do all we can to eliminate the possibility of this ever happening again.
I don't think I'll be posting much more tonight. I'm going to continue working to adjust and tidy things up around here, and then I'm going to get the heck out of this datacenter, go home, and get some much needed sleep.
I'm sorry about those things we have lost (mentioned above), but am thrilled that we have all our posts/threads/registrations.
I missed you all, and will be glad to see you all again.
Sincerely,
Jude
