Further Information on Second Life Issues, 1st-2nd October
Wednesday, October 3rd, 2007 at 7:15 PM by: Neuro LindenI’m Neuro Linden, one of the new team of Production Operations engineers here at Linden Lab tasked with assisting our support and operations staff with maintaining the health of the grid. I’d like to supply more information about the Second Life service problems of the past few days.
There were two primary problems: a change to the way we perform rolling simulator updates, and a hardware failure with the VPNs which allow distant Second Life servers to communicate. I’ll describe both of these failures in detail, below the fold …
We’ve recently changed the method used to deploy rolling simulator updates, in an effort to speed up and stabilize the process. Unfortunately, when we used the new system on Monday, it left many regions either shutdown, unable to report their status to our core systems, or running the old software version. The release team has identified several bugs in this new system, and are working to correct them before we use it again. Thursday’s rolling restart will use the old system.
So, after the new update method did its damage, and developers and operations staff were recovering the affected regions (at about 8pm PDT), a fault developed with two of our VPN servers. One was recovered successfully by operations, but at 11:30pm PDT, in the process of attempting to recover the second, it suffered a hardware failure, so we set out to replace it. While we were configuring the replacement machine, it suffered a ’stall’, which required us to intervene physically, and we weren’t able to get it up and running until 6am PDT. This returned our VPN to full service and most of the affected regions came back to life.
Some regions still had issues after this time, and our developers and production operations staff worked through the morning recovering them, with the bulk recovered by around 9am PDT.
We offer our apologies to all residents affected by these problems, and I would like to re-iterate Ian Linden’s previous comments that we are working on several projects to improve the reliability of Second Life, one of which is replacing our VPN systems with more robust technology.
October 3rd, 2007 at 7:27 PM
When will we have a single day where SL runs smoothly????????
October 3rd, 2007 at 7:34 PM
Thank you I look forward to this new process. No we don`t have a single day of “SMOOTH Server”.
October 3rd, 2007 at 7:35 PM
Could one or more Lindens please elaborate on how the new methodology for performing rolling restarts was (or was not) tested in simulation before being used on the main grid? The pattern that repeats here seems to be one of finding bugs in the production environment that millions of residents pay to access.
October 3rd, 2007 at 7:36 PM
Hahahaha #1
The answer is obviously no.
October 3rd, 2007 at 7:37 PM
I am unable to log on to SL.. I have been unable since 6:15 pm (pst)
Is this problem gonna be fixed tonight?? Am I the only one that is unable to log on??
Some answers would be appricated….
October 3rd, 2007 at 7:37 PM
Thanks for the update Nero, but what is gong to be done about all the packet loss and terrible lag since last update? The previous version had none of this, (well yeah we all get lag, but one hit on movement and I am 800m away????) little too much. I really like SL, so I want to try to be positive about outcomes. Will this new update tomorrow fix anyof this new lag?
October 3rd, 2007 at 7:38 PM
Wow thanks! That was a nice gesture coming out and saying it like that. I could’t upload textures or log in for long periods of time.. one of my sims went down for almost a full day. I was a bit more worried then usual.
October 3rd, 2007 at 7:47 PM
How about giving us a break for the man made breaks? Stop rolling out new anything for a very long time. Your back to the old pattern of destroying the economy weekly. SO STOP IT.
October 3rd, 2007 at 7:47 PM
Well I am a newbie to second life ..only three days old and already getting frustrated so I can understand how all the older members etc must feel …in the three days I have been playing I can never stay on line for more than about an hour or maybe two at most with out crashing …I do run a dual processor and have tried all the suggestions made for fixing crashing problems with no sucess ..maybe when you get stuff stablised maybe I can stay on line longer ..thanks
October 3rd, 2007 at 7:49 PM
Wow, just out of nowhere, a huge crash, and then trying to log back in, ‘Region is Off Line’ or something.
How about a little warning, please? I understand that SL is beset with issues, how could something this big not be, but at least let people know that you’re going to kick them off.
Please?
Thank you.
October 3rd, 2007 at 7:52 PM
we still can have sex right?
October 3rd, 2007 at 7:58 PM
There once was a time when Second Life was a small and close group. Then One day some linden decised ! ” HEY LETS TRY TO GET 10,000,000 USERS”.Well a short time after ( 1 year later ) There was no Second Life because it was hacked and left for dead” The End…….
Well lets Hope it wil be around longer then that. Too many people have Too much time and money invest on sl.
October 3rd, 2007 at 8:00 PM
to #11
Not according to my bed, no.
October 3rd, 2007 at 8:15 PM
just fix it get it done so sl can run smooth im so sick of it not bein done yet
October 3rd, 2007 at 8:15 PM
#5 Possible resent firewall settings change?
Everyone else, your utilizing a cutting-edge technology that is shaping the future of internet communications and utilization. If you prefer not to have sporadic issues I would suggest signing up somewhere else that uses old technologies, is barren of users and has no vision of the future.
SL is blowing away competition because they don’t just think outside the box.. there building it.
October 3rd, 2007 at 8:16 PM
oh and get rid of hippie pay and all that CAUSE IT DONT WORK IT SUCKS!!!!!!!!
October 3rd, 2007 at 8:21 PM
thanks for the ‘heads-up’ neuro… and thanks for the explanation of the problems that have plaqued us all these last few days… keep, up the good work!!!
October 3rd, 2007 at 8:23 PM
@15 ummmmmm I think.
October 3rd, 2007 at 8:26 PM
Thanks for the informative and detailed update!
Good luck with the changes!
October 3rd, 2007 at 8:45 PM
If it aint broke, don’t fix it. Hopefully you might grasp that idea some day.
October 3rd, 2007 at 8:46 PM
Also good luck with the changes. As we all know, computers have a mind of their own…
October 3rd, 2007 at 8:49 PM
Ive been having serious problems trying to get my 8 bots on to go camping. Please get this software fixed soon cause I’m losing tons of L$ not being able to log in 8 bots at one time….
October 3rd, 2007 at 8:51 PM
Well we appreciate the hard work and late night to correct this issue grid wide performance issues after a bad update you decide to try yet another new system,PLEASE listen residents want a stable grid,they want to buy in confidence,they want to be able to walk across sim borders and not die,If you tell us a rolling restart is due then ok we can plan around it we cannot plan around you testing a new system,Sort the grid then look at improving what is basically a 5 minute island hop.I do know you guys work really hard on advancing this platform but we need a platform everyone who meets specs can enjoy and i have sadly lost 3 long term friends in the past 30 days because they no longer want to be here 2 joined 2005,1 2006.
October 3rd, 2007 at 9:03 PM
I wonders why bother “mature” rating or no. Many time I be told is missing clothes me by friend. Screen self all look ok. Some other I no have clothes, some other has clothes me.
I teleports into new sim. Many time I need wait 3-4-5 minute before world rezz. If tries move before that I lags bad maybe to next sim or even more.
Understands you working hard infrastructure but this be getting worse. Needed laugh. Yesterday had question “Is you SL experience be get better or worse?” No could click any response.
October 3rd, 2007 at 9:04 PM
hmm traffic hits 30k+ we all suffer your hardware cant cope
October 3rd, 2007 at 9:11 PM
Your honesty is much appreciated Neuro. I’m a software developer. Nothing in this world is bug-free. Its just great that people are able to use this cutting-edge technology for minimal cost (or free of charge).
October 3rd, 2007 at 9:20 PM
[...] Further Information on Second Life Issues, 1st-2nd October I’d like to supply more information about the Second Life service problems of the past few days. [...]
October 3rd, 2007 at 9:23 PM
I noticed they took our ad money just the same, so they didnt feel all that bad the little bastards.
October 3rd, 2007 at 9:26 PM
Neuro
Thank you. It is refreshing to get some details. I’ve noticed in the last week that several of the lindens have been making the extra effort to let residents know what’s going on. Most of the residents that are complaining seem to be the unregistered ones who complain about everything.
Examples of this are the ones who can’t bother to look at the login screen. If it says there are other residents logged in then obviously the system is at least partially working so why bother posting an obvious question. As for the lag, It’s amazing that I can run this on my laptop from work on a wireless link with a 5k inventory and several scripted attachements in a sandbox with 10 or more users and don’t see the lag. Perhaps the unregistered residents who are camping should try going to a different sim.
I am not an apologist for the Lindens, but when they do communicate with us there are too many whiners. Instead of whining, write down your complaint and submit a bug fix. As for the one who said he was bounced 800 meters that is a communications server bug, (client to server so look to your internet connection). I’ve been working with Microsoft word (a simple word processor) since it was dos based and it isn’t bug free why would you expect cutting edge software to be bug free? Start reading before responding.
October 3rd, 2007 at 9:34 PM
You know, my experience the past month or so (after the power outage and near constant run of crashes and patches and stuff) has actually been pretty good. There were entire days when I could log in and not have problems. It’s gotten rough again the past few days, but at least I haven’t lost any items or money like before. Thanks for the heads up, Neuro, and I’ll be sure to buckle in on the 4th.
October 3rd, 2007 at 9:42 PM
Welcome Neuro, I usually like to complain about things, but it is a refreshing change to get an explanation of problems SL is having, and an awareness that LL is working to correct them, I have felt in the dark a lot of times about the workings behind the curtain and it often leaves me feeling like Linden Labs doesn’t pay attention, I like this blog, it makes me feel like I am a part of the process of SL, rather than a victim of it. Good work, keep it up.
October 3rd, 2007 at 9:45 PM
at ThaBiGGDoGG Richez, how about getting a job in sl instead of camping, it’s just a waste of sl…
October 3rd, 2007 at 9:55 PM
Is this “robust technology” thing will involve finally fixing that ‘object is missing from the database’ error that makes peoples’ items randomly unusable? You know, the one reported in http://jira.secondlife.com/browse/SVC-553
October 3rd, 2007 at 9:55 PM
Neuro, I am impressed and pleased with your honesty. Thank you, from a long-standing resident and outspoken critic. This is a very refreshing change, and I genuinely appreciate your candour.
October 3rd, 2007 at 10:07 PM
Since the rolling updates I have became a premium member. However I’m wondering how long that will last because since these updates my system periodically freezes and I crash once every hour minimum. Before them I could run Second Life for hours without a crash.
October 3rd, 2007 at 10:25 PM
#29
I just want a smooth experience without all the problems. Thanks for keeping us Informed about this issue!~
Hey Mike,
I appreciate the thought, but no, internet connection is fine, no other problems reported on my internet or wireless conenections.
I sent in bug report with snapshot of this incident. I had used my multipose bed which was 750m high, and I ended at 0,0,0 on the ground. Even took a picture! Most of the problems I am personally experiencing have been since the new update was installed on Monday. Up until then, my lag was very low, and now I am experiencing packet loss consistently in the 20’s. It is always redlined which wasn’t the case before. I am also not a *techie* just a resident who is trying to learn and understand the ins and outs. So when you talk mainframe,bits,bytes and values, and paramaters, too me it is BLAH BLAH BLAH
October 3rd, 2007 at 10:28 PM
Overall, I pay about $500 per month for this “free or almost free” service. The reason I do this is because SL is not just a game for me; I run a successful in-world business. Not surprisingly, when SL is experiencing down-time, instability, lag, broken tps, broken search, or even overall general “weirdness,” sales go down signficantly. This is a trackable trend that can be shown in a graph. So not only am I NOT receiving the service I have paid for, I am also losing income.
It’s all very well and good for LL to admit that they have a problem, though it is a bit redundant — we can see when there are issues. Having this confirmed is far less important than seeing some progress in solving the root of the problem. SL will not survive if LL continues to simply fix issues as they occur and then apologize for the “inconvenience” to their customers. I know of no other company who would find this acceptable, including other Online game platforms. Banks, newspapers, web hosting companies, game companies, airlines and more all have mission-critical hardware and software systems which they manage to keep running with relatively little down-time or serious glitches. Why is this such a challenge for Linden Labs? And more importantly, what is the company’s commitment to answering this question?
Let’s face it, guys — this is not just the developers’ sandbox any longer. It’s time to grow up and act like a real software company. Your customers expect nothing less, and will go elsewhere if you do not.
October 3rd, 2007 at 10:36 PM
Slifers,
Problems were, are, will occur again, obviously . This is IT.
I just appreciate the way Linden Team give us more explanations from now on. Thanks Neuro.
October 3rd, 2007 at 10:38 PM
im sure the team @ SL are sweating bullets when something goes awry.
its not easy and painless to do upgrades and configurations while thousands upon thousands of members are online.
i think the client software should have a tick enabler in the preferences to choose open GL, Direct X or software rendering though.
since the client seems to prefer openGL, i had to change my video card from an ATI to Nvidia (Nvidia works better with open GL, ATI works better with Direct X)
this would solve a good amount of “lag” problems, as some of the lag users are experiencing are from image rendering.
October 3rd, 2007 at 10:47 PM
I left SL about 9 mths ago after being a resident since late 2003. I left because of all the lag, the crashs every few hours or more, the griefers, the lack of suport for premium members (I was one)….etc, etc.
I came back in to SL after copius amounts of publicity on our Australian TV just to see if the new hype was true.
Guess what….same ol’ same ol’. I have been crashing every 30 mins or so in some sims, lag seems to be worse than before and still the dickheads are in abundance… Whats that saying….Once bitten twice shy…? or maybe Buyer beware….phud! Nero thanks for the info…but really after all this time , still Lindens have to put pen to paper to justify the unjustifiable.
October 3rd, 2007 at 10:48 PM
censorship dont you just love it
October 3rd, 2007 at 10:51 PM
Thanks Nuero, was wonderign what was happening,then realised new viewer!…well as one of your former critics, i would like to thank you for at least “speeding” up the info on issues. keep it up!
October 3rd, 2007 at 10:56 PM
People complaints of lot of thing about SL, me too of course, that’s good to pass nerves on something and let’s stress go off. let me say that: At the current Opensim development stade, In Linden Labs official grid, we can use attachements, hud, sit on prims and are not limited to a bunch of end-user useless mathematics functions and llsay as scripting commands. So all is not so dark
Come on Lindens, this post is a good examples of what to do, people need more communication from you.
October 3rd, 2007 at 11:20 PM
It would be nice if the game could be left alone for a while. All these upgrades and fixes just seem to make it worse and accomplish very little except creating new problems. Was wondering…at one time worked for a development firm, when we installed new programs and updates we ran a parallel system for a while….any possibility of that happening before just sticking it to us, come what may?
October 3rd, 2007 at 11:21 PM
Oh great, another “We offer our apologies to all residents affected by these problems…”. You guys cash in real money and now also deduct taxes - maybe you then should behave like a real company and not only apologize but stand in financially for your mistakes.
For now you not even manage do deliver a basic stability for weeks now. Even the odd web-based friends list is not working for days. Items keep vanishing due to your nifty little problems, inworld businesses loose customers and sales.
Come on, start to behave like a business….
October 3rd, 2007 at 11:29 PM
hmm 346$ i pay now for each sim thanks to vat i pay for maintenece of my sim it hasnt been maintained for 3 months how can i compeate with a no eu resident with a mainland sim he payes 195$ a month to you i have to pay 346$ how the hell is that fair i cant compeate lower eu members sim rates
October 3rd, 2007 at 11:37 PM
at least sex is good in the lag cum pending………. lmao
October 3rd, 2007 at 11:50 PM
Do people not grasp that SL is an ever changing, ever evolving thing? sure theres a few screw ups, but they fix them. if all the whiners knew how complicated SL actually was, they’d shut up and realise that LL is doing a good job keeping it running.
October 4th, 2007 at 12:01 AM
I’ve had a number of vendor failures over the last few days, really odd ones, like scripts unable to talk to the web, or the web unable to talk back.. it’s nice to know I wasn’t hallucinating. Thanks for the informative post.
Seriously, those of us who complain could easily become LL supporters, if you just avoided the corporate doublespeak and told us that problems were occurring and what you were doing.
October 4th, 2007 at 12:19 AM
Ann Otoole Says: “How about giving us a break for the man made breaks? Stop rolling out new anything for a very long time. Your back to the old pattern of destroying the economy weekly. SO STOP IT.”
Can LL please make a few isolated islands on their own grid to work the original Version 1.0,0 viewer and promise never to ever update again for those who hate bug fixes and new features. Then banish them there for their own pleasure.
I want bug fixes and new shiney, keep it rolling out before we get bored.
Thanks for the explaination Neuro, keep us well fed and we won’t eat you, it works for Torley
October 4th, 2007 at 12:25 AM
I had to laugh when I read this in the blog post announcing the rolling restart: “we want to wait until the service is in a stable state before we make any sweeping changes”. Obviously, there’s no point making it unstable if it’s already unstable. And they did deliver! I’m constantly amazed at how fragile the infrastructure is.
Second Life is a game and half the fun is wondering whether it will be working today. The blog is always my first stop before logging in. C’est la vie.
October 4th, 2007 at 12:27 AM
# 22 ThaBiGGDoGG Richez Says:
October 3rd, 2007 at 8:49 PM PDT
Ive been having serious problems trying to get my 8 bots on to go camping. Please get this software fixed soon cause I’m losing tons of L$ not being able to log in 8 bots at one time….
** THIS is the reason i deleted all my campers i had 4 to help newbies out and give back some to the community i deleted them all cus of these bot campers i hope more will follow suit as campers and traffic dont really do much anymore all are bots sad
October 4th, 2007 at 12:32 AM
please i just want one day were i log in not in noob clothing with boobs and able to TP to work i havent been able to tp anywhere out of the region for 3days and had stalling or crashing infact i had so much trouble i stayed away coming back on off chance it been fixed and finding out otherwise, Listen guys i paid my account up front in advance for a year and all SL done is cost me money than they add V.A.T and still give crap service FIX this mess and stop updating till you have a working FIX cos i feel you dont test it first in fact all of SL is your TEST BED…
October 4th, 2007 at 12:34 AM
I still can’t log into SL.. and nobody is saying when it is going to be fixed… I miss out on alot today… MONEY!!!! Because of SL..
Nobody is giving out any answers as to when many of us are going to be able to sign on again… Geez this is Ridiculous~!!!!
October 4th, 2007 at 12:34 AM
knitemarco0731 Tripsa Says:
October 3rd, 2007 at 10:25 PM PDT
#29
Hey Mike,
I appreciate the thought, but no, internet connection is fine, no other problems reported on my internet or wireless conenections.
I sent in bug report with snapshot of this incident. I had used my multipose bed which was 750m high, and I ended at 0,0,0 on the ground.
Reason is ur bed scripts are very unstable at that hight take it to around 300M *shakes Head*
October 4th, 2007 at 12:43 AM
It’s good to see that issues are BEING reported, rather than waiting for bug reports from the lab rats.
If this is part of the new changes, then confidence maybe restored for the long suffering and most welcome. We know life isn’t perfect, but being kept in the dark for so long, makes us feel like Mushrooms.
I sense a new wave of energy from the team at LL, lets hope this is a sign of things to come.
Great Start.
October 4th, 2007 at 1:05 AM
Ich werde das Gefühl nicht los, dass wir alle BETA-Tester sind und dafür auch noch bezahlen müssen. Ich wünsche mir einen Tag, an dem SL mal nicht abstürzt und an dem mal alles so funktioniert, wie es funktionieren sollte!
October 4th, 2007 at 1:09 AM
Lets just hope they keep focus!
October 4th, 2007 at 1:23 AM
Hi Neuro I appriciate it you come forward and explain what happend. But would it be really be to much to ask, that during these things someone posts that LL is aware of it
It was really bizare, the blog was celebrating that LL was doing a new agressive roll out, but at the same time the grid was having issues, with uploads, rezzing things from inventory and we could see concurrency drop considerable.
October 4th, 2007 at 1:32 AM
A VERY MEANT THANK YOU NEURO
For once and maybe even for the first time a LINDEN with B.LLS (the first one i ever read) … who dare to face the SL Citizens …. who has taken time to read all the comments (complaints) on hte blogs … and most important doesn t take us SL Citizens as little children. (What a refreshment)
Clear explanations …
Let s just hope that all will be corrected in due time ….
Again Thx very much Neuro “The Guy With B.lls” Linden
ps; Hope to read you a lot on the blog ….
*****
Und für unserem freund MARTIN WILLIS (comment 57) .. Steck dir die sorgen an den Hut freund …. SL ist wie SL ist … mit gute und slechte laune ….
October 4th, 2007 at 1:33 AM
laughing at #15.
what a dumdum.
October 4th, 2007 at 1:48 AM
Most webmail suppliers are deleting, or at least shutting down, accounts that haven’t been used in 1-3 months. Wouldn’t it be a good idea if the same thing would apply to SL? There must be hundreds of thousands accounts that are never used…
October 4th, 2007 at 1:50 AM
lol @ #15, really.
now to stay on topic:
I sometimes dont understand why people complain that much. I havent had severe packetloss in months now. I rarely crash (maybe once or twice every X days, and thats most likely because I work on heavy 3dsmax scenes in the background and SL runs out of video memory), and generally, the SL experience is pretty much stable for me.
while I agree that there may be people who have a really different experience, they should wait patiently and understand one thing:
LL is working on a MAJOR infrastructure rework, which will get rid of the central asset server and the rather unreliable VPN servers and replace all this with more recent and stable technology. do you REALLY think they are going to waste development resources on patching up holes now instead of rolling out the big project as soon as possible, especially when its the hardware infrastructure that is the major problem, and NOT some software bugs?
seriously, guys, read the blogs and THINK. theyve explained it long and extensively whats going to happen to improve the grid, yet you folks act like it never was said and we will use VPN until 2020 or something…
on other news, thanks for informing us what went wrong, I like extensive information like this rather than being left in the dark
October 4th, 2007 at 1:51 AM
Hey, I happen to agree with #15. And going around calling someone a dumdum just because you don’t share his/her opinion is oooh sooo mature, y’know. *rolls eyes* o_O
October 4th, 2007 at 2:10 AM
“”22 ThaBiGGDoGG Richez Says:
October 3rd, 2007 at 8:49 PM PDT
Ive been having serious problems trying to get my 8 bots on to go camping. Please get this software fixed soon cause I’m losing tons of L$ not being able to log in 8 bots at one time….
“”
Camping bots are a pain in the ass and shouldnt be allowed, theyre the reason me and a lot of other store owners won’t have campers anymore.
This upgrade is the worst for lag in a while, I cant move anywhere, freeze and keep crashing, can’t load or erit vendors or chairs. However, I know it will be fixed so I’ll just be patient. I have to say SL was running the best in a while on the last upgrade so why tamper with it??
October 4th, 2007 at 2:14 AM
Fattimah Collas Says: I still can’t log into SL.. and nobody is saying when it is going to be fixed… I miss out on alot today… MONEY!!!! Because of SL..
Nobody is giving out any answers as to when many of us are going to be able to sign on again… Geez this is Ridiculous~!!!!
So what, think of all the many more people who can’t PLAY!
Then again, there’s nothing wrong with SL, 30,000 other people are in, and the same people have the same problems for months, it’s got to click sooner or later the tinistst possibility that’s it’s not SL?Perhaps some people should get a RL job, get a real internet provider and buy a better computer instead of whinning they can’t run enough campers on their 20th century dinosaur.
October 4th, 2007 at 2:27 AM
Hi Neuro,
welcome to the team I’d say. Hope you will keep on posting what’s going on behind the scenes. It doesn’t really matter whether it is positive or negative news, as long as we know what is going on, it helps a ton of people otherwise left in the dark.
I’m looking forward to see the VPN’s move to a more stable solution, and while it will no doubt come with more downtime and regions being offline, the ultimate goal is a more stable environment.
October 4th, 2007 at 2:31 AM
Neuro, I see you’re new here. Welcome to the bear pit.
As an ambassador of the new culture of openness, can I ask you to go to the Lindens with a suggested change of working practice? When you have a planned change to the main grid, such as the new rolling restart method or the new sim code being rolled out, would the Linden’s be prepared to publish on this (or another) blog the testing procedures and results *before* putting the change on the grid?
Doing this would mean that (a) we can have confidence that the change has been properly tested and (b) that “many eyes” can see where additional tests could be added which might turn the unforeseen into the foreseen and avoided.
October 4th, 2007 at 2:32 AM
how about fixing the group issue that has been going on for months, your support dont reply to emails about it…
when an object is deeded to certain groups the owner shows up as (???) this stops our secure scripts working..
we have a lot of land and our partner has 2 islands, yet you dont reply to our support emails…
October 4th, 2007 at 2:33 AM
Nice clear factual explanation thanks
lots of growing pains, but it’s obvious we are growing and evolving - I feel a lot happier knowing *why* our user experience was changed - and what LL did to correct it, ie, replacing the failed hardware and resurrecting the failed vpns. Let’s remember what a complicated piece of stuff this is 
October 4th, 2007 at 2:36 AM
As a Web software developer with experience on some very large projects, I would say that Second Life shows poor quality control. The number of serious bugs that are uncovered AFTER going live with changes, is not acceptable by any well-run organization.
Some people here want to blame those who have not paid as being “whiners”. I do pay, and I believe we can all detect experiences that should not be happening, with major problems constantly appearing and disappearing, then reappearing with the next update of the viewer or network. It also appears they need high-end network consulting advice in order to avoid dependence on points of failure (such as the VPN mentioned) - too often a machine fails and it takes hours to recover. If set up correctly, it should be either invisible or require only minutes to recover.
It seems the company accepts too low a reliability as being the norm. A technology director with high standards should be placed in charge of all this, so reliability can improve. The fact that SL is still probably the leading virtual world in terms of maturity and number of members is not an excuse for poor quality control and technology implementations.
October 4th, 2007 at 2:50 AM
Thanks, Neuro.
Some days ago i was talking with some friends that the worse thing in LL was the lack of communication, all we understand (most of us) that SL as a system is quite complex, and is normal to have problems sometimes, but lately Lindens seemed hidding from the residents (no Town Halls, no Life Help, etc…).
I’m really happy with this post, Neuro, because it makes me think that things may be changing. I hope the rest of Lindens will follow this example.
October 4th, 2007 at 3:00 AM
@Martin
I have to agree — I know SL has a Beta Grid but then why is it, that upgrades, restarts, etc., introduce so many unexpected problems, not just occasionally but all the time?
And then, I often report bugs I encounter after such a “fix” (being a good little Beta tester!) but then when I read the blog on the next upgrade it doesn’t fix any of them. For instance for weeks now, when you click “edit appearance” it shows the back of your avatar. Two upgrades and three rolling restarts later it’s still not fixed.
So yes, why do I feel like we’re all Beta testers here? Beta testers, moreover, whose reporting isn’t certain to reach the people who can do something about it?
LL should thoroughly test everything on the Beta grid, where it has to run smoothly for at least three months before it’s inflicted upon the main grid, with the accompanying down time and related inconveniences for all users.
Also, yes, I agree: GET RID OF BOT CAMPERS. Keep the rule that says no alts for non-preems and limit alts for Premium to two. People should also be required to register their alts so that a ban to one will affect them all.
October 4th, 2007 at 3:18 AM
#1 … never
#9 … dont forget you have a first life too !!!
#16… yes please ban camping & sl will be a much better place…btw why the hell do people setup a sim and pay 2.5k to set a bunch of campers there anyway doh !!! oh yes i remember .. to get a false impression of there traffic count … so others would setup a shop there only to find out there only traffic is noobie and bot campers !!! yay for smart move, dance from happyness folks !!!
#22… read comment to 16 !!! doh or talk to All Gore, he will explain it too you better… raise your electric bills for 3 L$ per 10 min yes its worthed !!! doh you people are so silly !!!
#33… exactly… why the hell do we contsantly get the message (read tons) “Gesture is missing from database” & ” cant load Gesture please try again” doh while just getting online and doing nothing !!! LL fix that shit please !!!
#62… get rid of the non used accounts … put a non actif limit on them and send them an email after 6 weeks or 3 months to ask if there still want to login… so in the end the REAL WORLD will get an honest resident number… not like now … in every newspaper or story or blah blah they say Second life has over 9 mill users !!! LMAO
#71… thank you !!!
#72… to myself i will copy this post to be sure it gets posted this time… (bonus … my land is on sale VAT included)
To Neuro… goodluck mate
October 4th, 2007 at 3:18 AM
agree, get rid of the campers and restrict number of alts! And shut down accounts not used for 1-3 months, as the webmail providers do!
October 4th, 2007 at 3:27 AM
sorry LL double posts are the resul from the first post not showing up !!! dont blame the cookie, but feel free to delete this one and the first !!! Mwah /hug Lindens
October 4th, 2007 at 3:32 AM
Getting rid of campers and restricting alts seem like good ideas. However, is there any evidence that they cause the overall instability, bugs, lag, and so on? Possibly lag… but I doubt there are THAT many campers to explain the lag.
But the lag is probably not everyone’s #1 problem - it is the unreliability, loss of inventory at times, loss of business income due to downtime, etc. I do not personally have those problems, but I can see how bad it must be for those who do. It still seems to me that from the very highest level of Linden there must be a shift in attitude and methodology. Currently it seems that bugs are regularly re-introduced into the system and that point failures (single machine) bring down the environment for everyone.
It needs a steady hand, and support from the ownership of Linden, to start fixing these problems.
October 4th, 2007 at 3:34 AM
I will no longer be putting money itno second life until I know there is a stable protocol involved with secondlife.
Ive cashed out 2 weeks ago
October 4th, 2007 at 3:42 AM
I want to share my opinion here about the overal performance of both the Grid as they call it and Linden Labs. First a little history. I have been here since 12/2005. I pay my membership annually. I pay tier. More then some less then others.
Little more history. A number of years ago I ran a small but profitable hosting service. My clients ran 2D graphical chat services. The software that ran these services was at one time owned by Time Warner. The software came on most new retail computers along with AOL, Prodigy, Compuserve and other such commercial services. Despite the financial might of Time Warner there venture into the world of graphical chat went bankrupt. Enough history. Now to my opinion.
In order for me to provide these services I had to rent/lease servers. The company that provided my servers offered packages for as little as 99 dollars to up to several hundred. Though I was only paying $99.00 a month for my servers I could call this company 24/7 and after a couple minutes if that on hold be talking to a real live human. What was even better is this human with the help of support staff could solve problems very quickly. This company though now under a different name due to a merge hosts over 40,000 servers.
Now I see LL basically as a hosting service. Their performance compared to my previous experience with a hosting company is often pathetic at best. For 195 dollars a month in tier they should be able to off what most web hosts offer for 4.95 a month. 99% uptime. I understand that the software behind SL is a little more complicated then Apache or some other web service. Regardless Linden Labs fails to provide quality service.
I am sure there are many dedicated grunt level Lindens but I expect that the upper management has little regard for it’s customers. To me it seems they fail to listen and fail to care and fail to provide quality service. Perhaps they should reconsider name themselves all with the last name of Linden and take on various names. That is get out of the special group and think and act like those that are paying the bills.
In short it is my opinion that Linden Labs does not know how to to provide quality hosting services as well as many teenagers running hosting services that provide 99% up time from their bedrooms after school and on the weekends. I think frankly the Lindens have the take the money and run mentality.
October 4th, 2007 at 3:43 AM
The october started so nicely economy wise…Thanx LL for destroying this, however small trend. Who decided rolling out a new release to the month change was a good idea? Revenues are now pretty much dead again. Wonderful. I was just starting to forget about the illegal way of imposing VAT on EU customers…
Please don´t roll out more than once a month, please please
October 4th, 2007 at 3:50 AM
#1 … never
#9 … dont forget you have a first life too !!!
#16… yes please ban camping & sl will be a much better place…btw why the hell do people setup a sim and pay 2.5k to set a bunch of campers there anyway doh !!! oh yes i remember .. to get a false impression of there traffic count … so others would setup a shop there only to find out there only traffic is noobie and bot campers !!! yay for smart move, dance from happyness folks !!!
#22… read comment to 16 !!! doh or talk to All Gore, he will explain it too you better… raise your electric bills for 3 L$ per 10 min yes its worthed !!! doh you people are so ***** !!!
#33… exactly… why the hell do we contsantly get the message (read tons) “Gesture is missing from database” & ” cant load Gesture please try again” doh while just getting online and doing nothing !!! LL fix that **** please !!!
#62… get rid of the non used accounts … put a non actif limit on them and send them an email after 6 weeks or 3 months to ask if there still want to login… so in the end the REAL WORLD will get an honest resident number… not like now … in every newspaper or story or blah blah they say Second life has over 9 mill users !!! LMAO
#71… thank you !!!
#72… deleted
#76… deleted
#80…to myself i will copy this post to be sure it gets posted this time… (bonus … my land is on sale VAT included)
To Neuro… goodluck mate
October 4th, 2007 at 3:54 AM
Geeeee Just checked the agenda …. again a rolling restart at 1800 gmt …on this thurday
and this after a black monday , a darkgrey tuesday, a somewhat lightgrey wednesday
So Dear SL Brothers and Sisters, please join me in a prayer
Dear God Linden,
We all pray to you
so Second LIfe will be ressurecting from her ashes
like once did the Phoenix
We pray to you
so our beloved Second Live will no longer suffer
from buggs, downtimes and rolling restarts
We pray to you
So our AV’s would not loose any of their precious inventory items
We pray to you
So we can login again without any problem
AMEN
Dear Second Life Brothers and Sisters please give a very nice applaus for Neuro the only Linden who dare to come down from the Linden sky to talk to us ….
October 4th, 2007 at 4:31 AM
Once again, VPN is a problem child for this type of application. VPN packets are larger, there is a processing cost to encrypt and decrypt. Not to mention when traveling over the internet your connection can be perfect on both ends, but something in the middle that you do not control can be slow. To ensure QoS, which I would think be a permium you would want to have dedicated circuits between your server farms.
October 4th, 2007 at 4:40 AM
Excellent report Neuro. Professional and packed with rich detail.
Thanks a heap and a stack
Laz
October 4th, 2007 at 4:55 AM
Thank you for the update Neuro. Fact is that SL is still so unstable that even basic functions like logging in, moving, teleporting, making transactions or building are constantly hampered by freezes, crashes and all sorts of other glitches. Fantastic concept but really difficult to enjoy, neither for pleasure nor for business.
Pity.
October 4th, 2007 at 5:00 AM
Hi neuro
thanks for the update!! Its great we now know what is going on! We appreciate your post a lot! lol i like your name
I wonder if this trend of beeing opener will happen with some of the issues that could be more easily resolved than fixing bugs?
Like information about age verifications companies safety, mentioning the VAT to europeans before buying islands… things like that? (I know its not your concern.. just wonder… maybe a linden could be hired for that too..?)
@29: why do you assume the complainers are unregistered ones?
Its not just affecting the unregistered ones, these problems are affecting registered ones just as hard. Just take a look at JIRA and see many of the big names (shop owners of well known big shops in SL) have voted on issues which are mentioned here too many many times. They are not exactly unregistered ones!
October 4th, 2007 at 5:02 AM
I read recently that SL peaked a year ago this month and has since stagnated. I also read that the average person only logs in for 12 min. I honestly believe that the grid issues have everything to do with that. I would love to see great vast improvements before SL becomes a footnote in the virtual online experience. Don’t get me wrong I adore SL unfortunatly it’s become one very long commercial lately. Good luck with the newest havoc version. I am in a wait and see mode.
Mar/Cat
October 4th, 2007 at 5:08 AM
well al i can say is thank for keeping us up dated on the blog, but be nice to be updated in world before something that effect grid is used. so we have choice and can leave, if we feel we have a need to.
also LL had better what there backs goodle are working on there own metashere space (my world) and i know japnise are working on one too for the playstation 3, it gonna be called “home” and another japanise firm is working on another simmlar sl world opeing soon called “meet me”
so if LL dont fix bugs faster they might become underdog instead front leader.
Prepare yourselfs for competion LL the future migh be rougher for You!
October 4th, 2007 at 5:18 AM
because Ron Crimson #64 you and mr #15 don’t seem to get it. Second life is far from cutting edge it is actually old very old tech the software itself is nearly 6 years old the game has been open for over 4 that is not cutting edge. Not even close. This far into it they are having far to many issues. Sure programming is difficult but for the 2 years I have been here quality control is their biggest problem… there isn’t any!
October 4th, 2007 at 5:22 AM
…3…2…1…. cue the ranting… oh you already did! :-p
First off… pop quiz:
1. Do you know anyone who runs 16,000+ servers on a 24/7 basis?
2. Out of those 16,000+ servers (simulators) most are online most of the time - don’t you think thats impressive?
3. Now add in scripting, gravity, physics, building, prims, communication, web, voice, etc, etc - isn’t that even more impressive?
Anyone in IT *knows* you will have hardware problems, failures and software bugs, glitches and crashes. Nothing is ever perfect. Not Microsoft, not Linux, not Apple. When Word or Outlook crashes, you bang the desk, swear at Bill Gates and the restart it - you live with it day in, day out. Fact!
In SL it’s no different. Apart from the fact there are tens of thousands of servers, hundreds of databases and gigabits of bandwidth being used every second! You didn’t think your prims and house just “magically” stayed there did you?
One such glitch or hardware failure can have a *massive* cascade effect and effect many other things. Forgive the example, but look at the ripples when you throw a stone into a pond. Sometimes it’s not a stone that gets tossed in, but a massive building block.
LL do annoy me at times with their poor decisions - but these are not technology decisions, more “public relations” things. Technology wise, they really can’t be faulted. They are introducing backups and redundancy into more of their systems, but it takes time.
The moral of the story: It’s going to rain sometimes so carry an umbrella, enjoy the sun while you can!
Yours,
The XO
October 4th, 2007 at 5:24 AM
@ Arthur Fermi #82:
Bang on the money!! Well said
October 4th, 2007 at 5:30 AM
to some other posters:
Hey some of us do rely on camping and contests for Lindens, not all us are premium you know?!! an with service so poor from LL why should we bother upgrading? im still waiting for stability before i do upgrade;
Also it’s not like we dont then pass those lindens on in form shopping so u premium guys and girls do benifit in that way. think of us as money pump!
and basic accounts dont get alts! as far as I know. and gratful LL didnt end account after 1-3 months
if ever moved house, it takes BT an eteernaty with ISP to swich u back on! i was without a net conection for 5 months.
so e-amiling me wouldnt have been that effective now would it! ;-p
besides as some posts show some people go away and after long spell come back to LL second life
it up to acount creator to kill off there own Avitar! if dont want it.
and LL sory this took up two posts.
October 4th, 2007 at 5:31 AM
the problem is that it gets worse over time. if you stay on this road you are on atm, sl will behave like an early alpha version soon. it´s ok for a game, but not for a business platform…
October 4th, 2007 at 5:42 AM
We all know that problems can occur. Hardware fails, that missing EOL finally catches up to you. I am still amazed that LL has been able to scale up to the level it has.
There is a lot of good work here by LL. That said, the major problems, or perceived problems, truly distract from the achevments.
1) Paid Members perceive LL as being unresponsive to their concerns and needs.
2) Those of us who understand things such as version control, wonder if LL does. It certainly appears at times that they do not.
3) I realize that in some ways, SL is still and always will be a “Beta” environment. However, it often seems as though no testing is perfomred on updates of various types at all.
4) LL is getting cheaper by the day. Quit trying to make every penny now, and plan on creating a stable virtual world where you can continue to reap the bounty.
The way things are going: Gee, these virtural worlds that make up the internet are so cool. What ever happened to Linden Labs, the ones that really started it……
October 4th, 2007 at 5:46 AM
When will the “friends online” on the website be fixed?
October 4th, 2007 at 6:08 AM
Thanks for your detailed and informative post Neuro.
This is a refreshing change and I applaud you for facing the community and treating us like responsible adults - something that has been seriously lacking at LL of late!
@89 The XO. I couldn’t agree with you more! LL really CAN be annoying sometimes and its not because of their constantly innovative improvements to the grid. You’re right, its because of their poor ability in communicating with the SL community. Something that I, and many others in these blogs, have been banging o