[COMPLETED - 3:45 a.m. Pacific] We’ve just received word the scheduled maintenance is now complete and Voice Chat should once again be functioning normally. — Chiyo

[UPDATE - 2:30 a.m. Pacific] Voice Chat maintenance is underway. — Frontier

Inworld Voice Chat will down between 1AM and 5AM on Wednesday, April 16, for a major update in preparation for some additional voice services.

[COMPLETE 2 a.m. Pacific] The scheduled maintenance has been completed! — Frontier

[REMINDER] Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks. The maintenance will continue until 1AM Pacific Time. Please refer to the original blog post, Maintenance Events Scheduled, 13 - 16 April, for full details.

[Completed 12:07 p.m. PDT] Our system engineers have completed work on the Central database cluster and services have resumed.

We are about to undergo a 30 minute downtime in order to work on our Central database cluster. All services including transactions, logins and in world services will be affected.

We originally announced a Wednesday morning (Pacific) outage for the support portal.  That outage has been canceled.  All necessary work was completed during the first phase of maintenance.

[CLOSED 9:45 a.m. Pacific] The power cycling is complete.  All voice and live chat channels are now running at normal capacity.

*****

A minor hardware glitch has prompted us to shut down the power to one of our support teams for about ten or fifteen minutes. This is causing reduced access to voice and live chat for Billing and Outworld support; Concierge support is unaffected. This should take about fifteen or twenty minutes to address at most.

[Edit: in World Support channels are also unaffected.]

Resolved 5:30 p.m. PDT Logins, transactions and other in world services have resumed. Thank you.

Residents reported interruption of some services which included object rezzing, L$ transaction and login. This was the result of bringing some login servers online before they were ready. Services will resume shortly.

[UPDATE 15 Apr, 11:22 a.m. Pacific] The Wednesday maintenance for the support portal is canceled.  All the necessary work was completed during the Sunday night maintenance.

[RESOLVED 22:44] The service disruptions caused by tonight’s maintenance should now be taken care of. Thank you for your patience and understanding. - Twilight

[UPDATE 14 APR 22:27] The maintenance is causing some disruption of services for some residents, including logging in, teleporting and transactions. Please refrain from making any transactions until further notice. Please monitor this post for further updates as they occur. - Twilight

We have one maintenance event scheduled for each day between this coming Sunday and Wednesday, inclusive.

The maintenance is of two types, and each event will briefly impact resident experience in one of two ways.

Sunday 13 Apr, 9pm - 10pm Pacific: Our support portal will be off line for one hour for Phase I of a two-step vendor maintenance event. During that time, the support portal will be unavailable for chat or ticketing services.

Monday 14 Apr, 10pm - 1 am (Tuesday) Pacific: Phase I of an essential two-step upgrade to the breaker system at our San Francisco datacenter facility will require a shutdown of a number of our server racks.

Tuesday 15 Apr, 10pm - 1 am (Wednesday) Pacific: Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks.

Wednesday, 16 Apr, 7 am - 9am Pacific: The final phase of the support portal maintenance. The support portal will again be unavailable during the maintenance window.

Operations is already at work minimizing impact of the Monday and Tuesday maintenance by rerouting essential backbone services. However, we project that a total of approximately 1400 regions will experience three hours of downtime on one or the other of those nights.

Due to the way regions are dynamically assigned, we cannot predict in advance which regions will be affected, but as we get closer to the start times, we’ll publish any particulars that our own Ops team can control or pre-schedule to minimize impact on resident experience.

Resolved 09:06 a.m. PDT - Lotte Our database has recovered, and all services are working again. We apologize for the interruption.

Update 8:42 a.m. PDT -Kate Per our in world message, we will be experiencing downtime of logins, transactions and in world services. We expect this to be in effect for approximately 30 minutes and will update you with progress reports.

Our Central database cluster is struggling, and you might see timed-out logins, transactions and teleports failing as well as problem while rezzing objects. Our Ops Team is already working to resolve the problem.

[08:35 RESOLVED] The Database is good again, and all services are working now. - Lotte

[08:00 - Updated] We are still workin on the database, but the number of residents experiencing these problems should now be considerably smaller. - Lotte

[6:55 AM - Reopened] The problems from earlier have resurfaced. This is causing problematic logins and interruptions in various in world services for some residents. We are working now to find a solution and will update you on any progress. -Chiyo

[2:45 AM - Resolved] The database unpleasantness has been resolved and things should return to normal shortly. — Frontier

[1:00 AM - Update] No additional information or updates for the time being as we continue to investigate this issue.

There is currently some database unpleasantness causing interference with logins and a number of inworld services. Until further notice, please refrain from L$ transactions and rezzing of no-copy items. Ops in investigating the matter, and will hopefully some further news shortly.

[6:45 PM - Bumped] We’re still getting a few reports of rebellious objects which refuse to participate in object-to-object email.

If you are the owner of such an object, please visit https://jira.secondlife.com/browse/SVC-688 and comment with as much detail as you’re willing to share of:  The object name, the originating region/coordinates, and the destination region/coordinates.

[11:44 AM - Closed] This bit of detective work is finished, and our internal testing is successful. If you’re still having trouble with an older rezzed script which should be delivering interobject email, please let us know via the support portal. Thanks!

[09:36 AM - Update] We’re still working on the small group of objects which have stopped emailing properly. It will probably take us until early afternoon Pacific today to report significant progress or closure.

[02:43 AM - Update] A script has been run to go through each region and reset some email parameters for each one. Following this, new objects being created appear to be emailing correctly but we are still hearing of a few pre-existing objects not emailing at this time. We will monitor this situation overnight and aim to have an update in the morning.

[01:43 AM - Update] We believe that only around 15 regions are affected by this failure but we are actively working to resolve it. More news as soon as we get it.

We’re getting reports of object to object emails (e.g. vendors) failing across a small number of regions at this time. We are looking into the source of this problem and will update you in due course.