Nothing to see here. Move along.
"Brief" is just a way of saying it. The event happened from 9:15am through 3pm, at which time the site was up and running again.
What happened on that day? At 9:15am I removed one piece of software from the server and rebooted it. I didn't see the server up in under ninety seconds as usual, so I called ICONZ's help desk, which confirmed the server wouldn't boot, thanks to an error while loading the operating system.
The operating system was not happy with a change in drivers, and wouldn't boot without the original DVD for a recovery session.
And just this week I had contacted the ICONZ team asking if they had a library on site that could hold my recovery discs. But before I could send these up disaster happened.
While on the phone with the help desk I went to the Air NZ site and booked a seat on the 11am flight to Auckland - a 45 minutes flight, plus a 30 minutes taxi ride to the data center.
Because this server was originally installed from CD and all the OS since the original install were installed from disc files, we didn't have a DVD drive on this machine. I asked the ICONZ help desk to arrange for a DVD drive replacement to be installed and also asked them to commision a virtual server on their ICONZ virtual co-location platform, just in case things got worse. They arranged this while I was flying up to Auckland.
By 12:30pm I was on site and after evaluating my alternatives I decided to reinstall the OS instead of using the Symantec Backup Exec System Recovery image recovery. The reason for that is because the software that caused the problem would still be in that image, so it wouldn't be any good for me. Also, this server was running Windows Server 2008 Enterprise RTM as an update to Windows Server 2008 Enterprise RC 0, which was installed on Windows Server 2003 R2, which was installed on top of Windows Server 2003.
So a fresh install would make things easier - and cleaner.
Lucky the probem showed up during a restart, so the SQL database was perfectly safe, since the drives were still visible. Using the recovery console I copied the SQL database files to an external USB drive and loaded it on my test machine to make sure it was all perfect.
I then proceeded to delete and recreate the partitions on this machine - a small problem because those were dynamic discs with software mirror enabled and it seems Windows Server 2008 can't manage this well during the setup. To get around this I used a Windows Server 2003 bootable DVD to delete and create fresh new partitions.
With this sorted, Windows Server 2008 installed very quickly. Installing SQL Server 2005 was not a problem, and soon I had the Geekzone site up and running.
The next steps were to reload some extra software on this machine, apply the Windows Updates (which required lots of reboots due to having SQL Server 2005), and configure the Symantec Backup Exec System Recovery on this newly built machine.
By 6pm I was in a cab going back to the airport to fly back to Wellington. I spent most of Friday doing the last bits of configuration required, testing the backup routines to make sure everything was ok and so on.
The great thing here is that we had no data loss at all. We also had a daily backup to an external drive, plus a weekly backup to an off-site location via FTP. And yes, I do test the backup contents to make sure they are valid (and so should you).
I've installed Symantec Backup Exec System Recovery to replace Acronis True Echo Server, as part of a review and I am really impressed. It's easier to manage and has a much better disc space management consolidation routine, including backup rotation, storage consolidation, etc. And it also allows conversion of image backups to both VMWARE and Virtual Server formats. I didn't have to use it because of the decision to rebuild this server, but it is now running full time as the main backup routine.
A complete image backup is done and scheduled for daily incremental backups. The daily SQL Server 2005 backup is running, as well as the transaction log switch. The weekly backup to the off-site location is back in place and all software is up-to-date.
I will be recreating the recovery DVD images from this new system and shipping those to ICONZ so that we can have a faster recovery next time - without perhaps having to have me flying there.
A big thanks to the folks at ICONZ who arranged the extra hardware and commissioned the virtual server - which we wound up not using at all.
Also thanks to all the Geekzone users who contacted me - a few voice calls, some SMS, inumerous Twitter messages and support from the people in our #geekzone IRC channel - including the offer to drive around Auckland to ferry any hardware if needed.
Lucky Geekzone shows up in the light sites list as #10 with a 97.2 kilobytes payload (not counting images) and eight JS files (mainly Google Analytics, Nielsen//NetRatings, Statcounter). The heaviest page in the list is vouchermate.co.nz site with 1.3 MB!
Check Jonh's blog post for lots of good information!
The idea is simple: you have large files and need to safely transfer these across your network or the Internet. You might even want to have your users to upload by themselves, but have their files going into different folders so that they are completely separate from each other.
In my case I needed to transfer a couple of large (20 GB each) virtual HD files back to my home server. Those VHD were created from an image of the Geekzone server drives so that I can have a copy here I can use for testing - and backup.
Blade Tranfer Services is very easy to install - it will automatically configure itself as a folder on your IIS server and start. You will just browse to /bladets and be greeted with a page with direct links to upload and download applets.
There is also a full client, but the web-based service is very good. You can pause, restart the uploads and you know for sure the files are 100% transferred - no corruption, no loss of data.
You can also enable a /drop folder so people can browse the upload folder directly - but the download option is much safer.
It uses http and https to transfer files and a configuration page allows you to change the default upload folder on your server and SMTP and SNMP notidications.
You also know that this is only one of 31 notebooks being given away by 31 websites, in a promotion called the 31 days of the dragon.
I read reports of the first notebooks being already awarded, so you should look at the list and enter the competition on each one of the sites, following the schedule.
We will be running our competition from 29th May through 5th June - this is the period you will be able to enter the competition on Geekzone to win one of the 31 HP HDX Dragon packages.
Before then, check the other sites and enter their competitions - you can win in any of the sites!
The difference is that people following the Project Niu know exactly where it is, all the time.
You just have to point your browser to their Where's Niu? map page to see where this floating device is being taken by oceanic currents. The Niu was set off of the Hawaiian coast on this trip, and the group plans to take it to other regions later.
Twenty successful teachers will each be given a $1000 cash grant to spend on a programme or project of their choice.
The programme is open to all fully registered teachers working in New Zealand’s state, integrated and private primary and intermediate schools (Year 1- 8), of any decile and location.
Submissions are accepted until Friday 9th May 2008.
This is an internal conference for their technology people. Really good attendance here.
Tonight we have a special dinner, including Telecom and Gen-i CEO Dr Paul Reynolds - and yes I am still trying to get an interview with him, but more on this later.
If you are attending this conference look for me around the floor today and tomorrow.
Basically it is one year of free data for POP/IMAP and Exchange ActiveSync - while you are in the 24 month contract and using the Okta Agent - you can check our Okta Agent review here.
You can also use this data for downloading attachments from your e-mail server and of course a bit of light browsing - perhaps a link you receive on an e-mail for example.
The device is not to be used as a modem for laptops, servers, etc - and a fair use policy applies. For what I've heard this will be monitored, but most people using POP/IMAP and Exchange ActiveSync would be pretty safe. This is from their FAQ:
Some examples of where your usage may be considered excessive and/or unreasonable are if you use your OKTA Agent:
- For sending and receiving extremely large email attachments like TV shows
- To watch TV programmes or download other streaming media - video or audio
- As a modem to download email or surf the net on your home computer
- For peer to peer data sharing (BitTorrent for example)
All FAQ and Terms and conditions are in the Okta Agent Free Email plan page. This is available from today, although you may not see anything in stores until next week - but you can ask around, the channels should know all about it now.
Great idea Telecom!
Ponoko was launched at the TechCrunch 40 conference bac in September 2007, but the guys have been working on that for a couple of years.
Basically it is user-generated product manufactuting and distribution. Really cool idea.
Here is part of their press release sent out today:
Ponoko gets greener with U.S. global head office
New Zealand company Ponoko today announced the establishment of a new global head office and manufacturing facilities in San Francisco as the website gains traction among U.S. users.
“Being able to make products on-demand, close to where people live, reduces waste and cuts down on the carbon emissions associated with transporting products to consumers. Our facilities in San Francisco mean that we’re starting to see this become a reality in the United States, and the appointment of Graham to our board of advisors is a huge endorsement of Ponoko’s vision for a more sustainable approach to the way goods are created, made and delivered,” says Ponoko co-founder and CEO David ten Have.
Ponoko is the world’s largest marketplace for product plans. The website works by connecting designers with consumers to share, buy, sell and make these designs into individualized goods, on demand.
Since launching, Ponoko.com has attracted a wide range of product plans for items ranging from household items and accessories to jewellery and wearable art. Today Ponoko also launches a new website that showcases these designs, and enables consumers to purchase products and plans directly from designers and makers by credit card or PayPal.
“The new website features make it even easier for people to buy and sell truly unique products using Ponoko,” says Ponoko co-founder and Chief Strategy Officer Derek Elley, “and continues the growth of a new class of business people using the Internet to design, sell and make digital product plans into real goods, on demand.”
They say over half of the U.S. visitors to Ponoko.com are from California, moving to San Francisco would be only logical.
Good luck guys!