maandag 29 december 2014


Update 2014-12-28

by allenkerenskyosg
We hope you had a happy holidays and a merry Christmas, Boxing Day, or other time of celebration.
Despite the holidays and large case of Real Life Interference, some progress was still made this week.
DRIVE RECOVERY STATUS
We expect the data center to receive the recovered drive once the recovery service and data center staff return from holiday breaks in early January.
ASSET SERVER STATUS
After numerous attack attempts were detected against both asset servers, additional extensive hardening steps have been taken, delaying database and asset server work. In practical terms, almost 200 recommended security configurations on both asset servers were checked and reconfigured as necessary including full patching and verified reboots.
OSGRID RESTART
Two major steps remain to restart OSgrid:
1. Start the database replication
2. Start the asset server
Both of these steps require a number of specific reconfigurations and new tuning from the previous OSgrid configuration, which require a number of deliberate discussions before each step of the implementation.
OSGRID RESTRUCTURING
No discussions on this were taken during the holidays.
We hope you have a safe and happy New Year's Eve and we're all looking forward to a great new year for OpenSimulator and OSgrid!
Next Expected Update 2015-Jan-04 or 05.

maandag 22 december 2014


News 2014-12-22

by allenkerenskyosg
DRIVE RECOVERY STATUS
Recovery service completed drive file conversion and restore to a new disk.
Recovery service has been paid with thanks for their tenacity in dealing with such a large and unique dataset.
Caveats:
Recovery service cannot guarantee OSG can use the data once restored to asset server due to various conversions required to recover data to this point.
OSgrid cannot verify restored database and files can be used for asset service for the same reasons.
Next Steps:
receipt of recovered disk
copy to temporary grid asset server
test restart temp asset server
validate temp asset server
plan next steps based on validation results
ASSET SERVER STATUS
Asset Servers 1 and 2:
Hardware active
RAID configured
OS installed
stock database replaced with higher performance database
OS administrivia like additional loadout and hardening
RAID card battery replaced in asset server 2
Next Steps:
configure database replication
Install new asset service
Begin testing
Additional database tuning
OSGRID RESTART
There is NO estimated time of arrival (ETA) for public re-open.
Test restarts for asset server qualification have begun with initial database configuration and tuning.
That being said, obviously progress is being made to that end.
s soon as we can identify an ETA, we will let you know here.
OSGRID RESTRUCTURING
Due to this crash, everyone agrees changes are necessary to infrastructure and operations.
The restarted OSgrid will include changes to asset servers and plazas to accommodate the needed changes and improve redundancy and reliability.
More news on restructuring changes as these item-by-item discussions reach decisions.
Next Update 2014-12-28 or 29.
allenkerenskyosg | 2014-12-22 at 4:33 pm | Categories: Disruptions & Downtime | URL: http://wp.me/pSI0n-9j




maandag 15 december 2014

Recovery News 2014-12-15

by allenkerenskyosg
RECOVERY
Recovery is complete.
Recovery service cannot guarantee OSG can use the data once restored to asset server due to various conversions required to recover data this far.
OSgrid cannot verify restored data can be used for asset service due to conversions required for recovery.
Pending: payment to recovery service - receipt of recovered disk - copy to temporary grid asset server - test restart temp asset server - validate temp asset server validation - plan next steps based on validation result
ASSET SERVERS
Asset Server 1: Hardware active - RAID configured - OS installed - Database installed
Pending: replication - Install and configure ROBUST asset service
Asset Server 2: Hardware active - RAID configured - OS installed
Pending: Database install - Replication - Install and configure ROBUST asset service
GRID RESTART
There is no ETA for public re-open.
Test restarts for asset server qualification are planned within the month.
RESTRUCTURING
Due to this crash, everyone agrees changes are necessary to infrastructure and operations.
The restarted OSgrid will include changes to asset servers and plazas to accommodate the needed changes and improve redundancy and reliability.
More news on restructuring changes as these item-by-item discussions reach decisions.
Next Update expected: 12-21 or 12-22.