[Resolved at 10:50pm Pacific] Our Support Portal is back online!

[Updated at 10:10pm Pacific] Maintenance is in progress. — Frontier

As reported earlier this week, our support portal will be offline for system maintenance tonight, Saturday, 26th April.

Our software supplier has reduced the length of the downtime from six hours to three, from 9:00pm-Midnight PDT.

During that time, the support portal will be unavailable for chat or ticketing services.

Apologies for any inconvenience this may cause to you.

[UPDATE 25 April 08, 3:40 pm Pacific]

Our payment processor has fixed and tested the process.  All card operations (including updates) are now running within normal time frames.

*****

Due to an issue with our payment processor, we are unable to update your credit card information at this time. Attempting to delete and update your credit card will be indefinitely delayed. You may enter card data but it will not be active in our system for potentially several more hours .

If you have valid credit card or payment information, you can continue to USE it with no problems- this only affects updating card information, or initial card data entry.

Updating Paypal payment information is completely unaffected.

We are working with our payment processor, and credit card updating will work as normal as soon as their issues are resolved.

Our support portal will be offline for six hours of system maintenance this Saturday, 26th April from 6:00pm-Midnight PST. During that time, the support portal will be unavailable for chat or ticketing services.

As originally reported here, our auction pages will be offline for scheduled maintenance this evening from 9 pm until 11pm PST.

We apologise for any inconvenience this essential event may cause.

[2:43 AM - RESOLVED] The database is back and transaction histories are now viewable again. Thanks for your patience whilst we sorted it out.

Residents will find that transaction histories are not showing up when running a query on our website. This is due to an error on one of our slave databases, which is now in the process of being resolved. It’s hoped that this will be back in about 20 to 30 minutes.

Sorry for the inconvenience meantime.

[10:52 PM Pacific --teeple]

Operations has just addressed an unexpected performance hit on the central database cluster.  For a very few minutes, logins were slow or stalled, and various in world functions such as land and L$ transfers and map/search lookups were inhibited for many residents.

We were just finishing a blog post to report the service disruption when Ops found and corrected the root cause.

In addition,  Ops has corrected a misconfiguration one one of our two new firewalls which was causing timeouts on approximately 50% of attempts to hit our homepage for the past few hours, as well as inhibiting some percentage of inworld LindeX orders and land transfers.

While inconvenient and frustrating, we believe that all stalled transactions resulting from the two gremlins this evening have reverted harmlessly…that is, that no residents have suffered incomplete transactions.  If you have experienced a loss this evening, please contact us via the support portal.

Thank you for your patience, and we apologize for the inconvenience.

Resolved 3:38 p.m. PDT The support portal is up and running.  We look forward to assisting you.

Update 3:14 p.m. PDT Our support portal is available. Some residents are receiving a 500 error message when attempting reach support. We will have this corrected as soon as possible.

Our support portal is currently unavailable. Residents might receive a 500 error message. We are on top of this and will have the support portal up and running as soon as possible.

[2:30 PM --Chiyo] It is taking a while for the system to recover from the stress of today’s earlier problems. That means for the next few hours you may experience intermittent problems with teleports, rezzing objects, scripts, transactions, appearance etc.

This also affects many regions that are still offline and have not restarted as of yet. The current stressload on the database from everyone’s return is delaying this process. Please be patient as things work themselves out to return to normalcy. This process may seem slow at the moment but it is progressing as fast as we can make it happen. Thanks : )

[12:55 PM --teeple] We’re open. The outage has been successfully addressed by our service provider. Traffic has been diverted around a faulty router within a major internet carrier’s facility. Please be patient logging in for the next few minutes until the initial surge of logins has been processed. If possible, ride the login process out, rather than quitting and restarting, in order to keep your places in the login queue.

Some regions are still being returned to service, and Operations and Concierge will be working with those as quickly as possible.

[12:06 PM --teeple] Our upstream provider is continuing work to resolve the outage.

[11:01 AM --teeple] As Jack remarked earlier, this is an extraordinarily prolonged and unusual disruption. Many high-profile services and corporations are feeling the pain this morning, and full scale efforts are underway to isolate and fix the root cause.

[10:11 AM] We are still trying to put our fingers down on the cause for the problem, please bear with us a while longer! We apologize again for this interruption of your weekend plans!

[09:07 AM] At the present time, in order to resolve the current operational state of Second Life, we will need to force log out all users currently still logged in. We apologize for the inconvenience and hope to have the issues resolved as soon as possible. Updates will be posted here. -Lotte

[08:30 AM] Our Ops team is still working with the technicians from our ISP to fix the networking problems. Stay tuned for updates! -Lotte

[07:30AM 04/05/08 REOPENED] It seems the problem is even harder to nail down than we earlier supposed. We need to close logins again to go back to diagnostics. -Lotte

[3.30 AM] [RESOLVED] - The doors are open and we are back. It turned out there were multiple issues with our ISPs network which have been worked around for now. The network provider is working to resolve the issue permanently - Matthew

[2.30 AM] Folks, it’s been a rough night so far and we apologise for the lack of service right now.

Like most network providers today, the ISPs who provide bandwidth for Second Life actually bundle many network links into large virtual links between their data centers. Traffic for many customers then flows across these bundles. Starting at approximately 19:30 PST, some of the special routing that handles aggregation of these bundles at the ISP level malfunctioned causing us severe packet loss of over 50% on the portion of traffic going to Linden Lab’s data centers. We had no option but to disable logins.

Since this is a highly unusual failure, it’s been complex and time consuming to diagnose. We are working with our network provider’s engineers towards a solution and will be back up just as soon as we can.

– Jack Linden

Resident logins have been disabled until our toplevel routing issues have been resolved.

As originally reported here, our auction pages will be off line for scheduled maintenance this evening from 8 p.m. until Midnight Pacific.

We apologize for any inconvenience this essential event may cause.

Our auction facility will be off line for four hours of scheduled maintenance next Wednesday, 2 April, from 8pm until midnight Pacific time.

We apologize for any inconvenience this event may cause.

– teeple