More info on this week’s grid problems.
Saturday, August 11th, 2007 at 4:59 PM by: Ian LindenFor the curious, I want to shed more light on the causes of the stability problems Second Life has experienced over the past four days. First, some background: the two facilities which currently make up the grid use encrypted IPSec tunnels (a form of VPN) to securely communicate with each other over the Internet. This is what allows, say, a simulator in Dallas to query our databases in San Francisco. Needless to say, if these tunnels don’t work, SL doesn’t work very well either.
On Wednesday, we experienced a failure wherein our primary bandwidth provider started dropping about 95% of our IPSec traffic bound from one site to another, but only for some of the tunnels. Other, non-IPSec, traffic was largely unaffected, and so this was a very difficult problem to diagnose and, crucially, get the provider to fix. While waiting for that, we started moving banks of simulators onto unimpaired tunnels. Once a good percentage of the SL was back up, we enabled logins, but didn’t get all the sims back up until the provider fixed the problem.
At 3AM on Thursday, we had trouble again and two failures actually overlapped. Our provider started dropping traffic again, with a different set of tunnels affected, and, around the same time, our largest tunnel failed entirely. This is apparently due to a bug in the router model we’re using, which the vendor believed was fixed but which, it seems, wasn’t really fixed. I still don’t have all the answers on that yet, but we’re now in the process of taking those routers out of service. It was several hours before the bandwidth provider fixed the traffic problem and we were able to get the failed routers back online.
While all of this has been going on, and still today, a serious bug in the Second Life software is causing 5-minute lock-ups of the communication channel between simulators and our various databases. This is a difficult bug to describe, but essentially, we see a very tiny percentage of database queries stall indefinitely after returning their data. Our software notices after five minutes and re-starts the database client, but by then some number of queries have usually been lost. This causes affected sims to occasionally (or in a few cases regularly) experience inventory problems, search problems, transaction problems, the works. Without any code changes, the bug seems to have gotten more common over the past week; on Friday evening the engineers working on the problem identified a potential fix, which we’ll be testing over the weekend.
Finally, there was an unrelated problem Friday night: a disk drive in one of our databases failed in a peculiar way, which caused it to stay in service but perform far too slowly to keep up with load. (Normally drives which fail are automatically shut down by our RAID systems.) Logins and some residents’ inventory remained unreliable while we diagnosed the problem.
The end result is a pretty rough week for Second Life Residents, and I can only offer apologies for that. The good news is that we have many projects in the works which will help improve the reliability of SL. We’re working on new infrastructure which will help us withstand the failures above, as well as store and transmit data more reliably, and we’re working on reducing both client and server crashes. Several of these projects are new, thanks to our growing technical team. We’ll share more information on these efforts in the near future.
August 11th, 2007 at 5:06 PM
Thank you for trying to keep us in the loop on what’s happenening ‘behind the scenes’. I think you all are doing a remarkable job when I consider the shear volumn of data you’re dealing with in real time environment.
August 11th, 2007 at 5:07 PM
Thanks for the details. Reassuring to hear that it just seems to have been a bad week rather than some deep rooted problem. Fingers crossed that normal service had now been resumed…
August 11th, 2007 at 5:08 PM
Glad you found the problem. I, and others, I’m sure, will be eagerly awaiting the fix.
One question, though. Will this fix the strange vehicle behavior? Because when I’m riding a vehicle, if it gets up to too high of a speed or goes into just the wrong area(Not an unreasonable speed or an area I’m not allowed in), it spins out of control, crashes into(Read:Through) the ground, or just starts accelerating out control until I relog, and I’m having problems getting anywhere, or even finding places where I can rez another vehicle. So, will the fixes address that?
August 11th, 2007 at 5:10 PM
Does explain a lot….would like to know which version this bug was introduced in, pertaining to the bug with the DB queries.
All in all, i think this explanation makes a lot of sense, holds together with the facts on hand…..keep up the good work on working around the unlucky problems and everything, and i hope the fix that’s being worked on over the weekend works ^.^
August 11th, 2007 at 5:10 PM
Thank you so much, Ian. I was really hoping that you guys would be able to track down these issues, quickly, but I don’t live on hope. I know full well how complex this system is, and things can go wrong in ways that overlap and interact to make it very difficult to fix.
For the most part, SL seems to be behaving as well as it has in a long time, but I’m sure you’ll get the usual rash of complaints. I, for one, am happy that the Grid is as least functional, again.
August 11th, 2007 at 5:12 PM
Well atleast someone sheds the damned light on problems AINT THAT GREAT!!!!! W00t W00t!!!!
August 11th, 2007 at 5:14 PM
First time bothering posting in a blog.
What a week its been, loss of inventory, loss of L$, inability to log in. At this time im getting the message “Login failed. Unable to connect to a simulator” So, for me the log in issues arent fixed.
The problems over the last week are recurring problems we have all seen countless times before, normally they rear there ugly head again after a major patch.
So will this “fix” thats being tested this weekend, be a fix. Or just another bandaid that will fall off in short time.
My faith in LL is as stable as the grid.
August 11th, 2007 at 5:14 PM
time will tell…time will tell.
August 11th, 2007 at 5:16 PM
Wow - It’s great to get some info on this! It HAS been a tough week, but it’s great to know someone out there is working on it. Way to go and keep us posted.
August 11th, 2007 at 5:18 PM
Hang in there ya’ll, Some of this info is over my head, but I know you are trying
August 11th, 2007 at 5:19 PM
thank you Ian for letting us know the real scoop and treating us like adults.
all though i hate the down time i can understand and emphasize with your plight.
again thank you for the real scoop.
August 11th, 2007 at 5:20 PM
Explanations like this are needed to keep people (myself included) from jumping to conclusions. It’s good to hear such a thorough explanation… helps to reduce the rather large and jaded opinion pool out there that consists mostly of you guys not knowing what you’re doing.
August 11th, 2007 at 5:22 PM
So what are you doing actually of me getting back to the AV is was before? Since your latest grid problem that so said was fixed made me end up nude not able to edit my appearance while my clothes are loading. The are downloading whole day now. Still no fix! Instead of portraying like all is well! fix your grid and let me please enjoy something i paid for. I didn’t pay to walk like a nude grandma. I can’t talk to my customers like that and can’t open my new club looking like that!
Clear cache, ethernet connection, reinstalling second life, restarting the region still leaves me naked!
When will we get compensated for all this? I pay monthly 295 USD to end up in all kind of problems just because you can’t manage to provide what you want the whole world to believe you deliver.
Why did i till know never received a version of the SL client that would just work fine? Why is that? I can imagine some versions end up bad, but all?
August 11th, 2007 at 5:23 PM
no the truth is that this voice shit creates more lag and is ruining everones fun, make a new viewer 1 with voice 1 without, that both are live… i cant stand this, you took off the votes stuff that i put in about 5k worth of lindens in on the votes and now you FORCE everyone to have this voice crap on their viewer ALOT of people do not like the idea of voice
August 11th, 2007 at 5:23 PM
Ohmygawd, as you were describing what had gome on over the last few day I couldn’t help wondering if you weren’t being given a sign from GOD that you should stop trying to do his job and stop being the morals police for SecondLife.
Yeah, His thumb may be hoovering over the BIG RED BUTTON right now…ready to Zzzapp the crap out of something else until you get the message. I suggest you start some good clean living and offer up a sacrifice of some kind, like maybe giving up VOICE or paying back the citizens for all their lost inventory. Remember, it ain’t a sacrifice unless it hurts a little.
August 11th, 2007 at 5:23 PM
One very, fundamental, basic change would help a lot… make the bloomin’ database transactions frikkin’ *ATOMIC*. It will prevent lost inventory from failure to rez, PERIOD…
August 11th, 2007 at 5:26 PM
This is just bad news! Again new stuff while SL still works like crappy software!
August 11th, 2007 at 5:26 PM
I like how everyone get new feature but looses all there stuff!
August 11th, 2007 at 5:28 PM
“”"”"”"”"”"which the vendor believed was fixed but which, it seems, wasn’t really fixed.”"”"”"”"”"”"
from what i do know that ??
August 11th, 2007 at 5:28 PM
I belive Sl is to big and to advance for current Tech. or the internet is?
August 11th, 2007 at 5:31 PM
Well, as much as I detest LL and the pile of crap SL has become, I must say this is FINALLY a step in the right direction… Some HONESTY and OPEN COMMUNICATION instead of just taking people’s money and pretending it’s “none of their business” why things are in the state they are in. It doesn’t excuse the mess and it doesn’t make things any better for paying customers, but at LEAST it seems maybe some of the arrogance is fading… keep heading in that direction and I might even lift my curse….
August 11th, 2007 at 5:33 PM
Agree with Solomon. The Het-Grid is here; why are you still playing with Voice and WindLight? You can change the way the grid talks to itself and the asset server now! Why isn’t this happening? And if it is, PLEASE tell us that it is! You need to slow down the development of WindLight, and get some of the people working on that moved onto the problems. Once the asset server does exactly what it’s supposed to 99.99% of the time, then we can talk about pretty skies.
Regarding the weeks problems; it sounds like your vendors and providers owe you a refund or two. Good luck keeping tabs on the external problems, and even better luck with fixing the internal ones.
For all the Lindens and the Grid, Cheers.
August 11th, 2007 at 5:36 PM
Ian, this is a great post with technical details as to the outages. It seems like only one of the issues, and not even one that caused the grid to close, resides in the realm of the “code” itself- everything else is vendors. In that, it’s a stroke of bad luck, but a good thing to know. I’m expecting a rolling restart or client update next week for that bug fix
August 11th, 2007 at 5:38 PM
ty for keeping us residents up 2 date thumbs up and keep up great work !
August 11th, 2007 at 5:41 PM
This explanation is all well and good but maybe fixing and stabilizing the existing product before adding features that cause a drain on the system. We are loosing inventory that we can not recover loosing sales because items are not being delivered and there is no way for us the paying customers to recoup our loss due to system wide outages or shortfalls. Granted one or to inventory items can be tolerated but when over time you are loosing inventory that starts to amount to a real cost what si Second life going to do say sorry but we cant help. I appreciate the explanation but is steadily loosing confidence in the program as a whole especially considering the cost of owning a island and trying to operate several business ventures in world. Fix what you have already before introducing any new features that open the doors for more stress on the system. Hope this post doesn’t get deleted and your willing to listen to criticism as well and you so ready to take praise.
August 11th, 2007 at 5:41 PM
Hey Guys… stop whining…
Shall I come and mess with your tiny lil home use router?
A lil falg… a lil click and you surf like 10 years ago on your phone line.
Thank you LL for that great background info - that makes absoltely sense.
Seriously… stop whining and complaining here - slowly it’s getting old.
Add serious comments - serious critics which helps LL to find the bug.
And not about how the casions went down the drain and camping chairs pay less.
Hell I never used a camping chair in my whole SL.
Just think:
Power consuption of a PC - compared to what you get with camping…
Hehe
Did I mention that I love my Platic?
Live long and prosper
August 11th, 2007 at 5:41 PM
over the past 4 days?
August 11th, 2007 at 5:43 PM
hey what ever it takes get it fixed. thanks for the heads up on what is going on.
August 11th, 2007 at 5:44 PM
This build and client is pure poop. Crashing all day. Im at crash #23 for the past two hours. Your CEO needs a reality check and engineering should be ashamed of this spaghetti code that is terminally flawed.
Voice was a horrible idea and was implemented poorly. Most of those who wanted voice can get better results in using Skype and still active in-world. Please become a publically traded company so we can demand resignations.
August 11th, 2007 at 5:45 PM
Dear lord. Someone send these poor folks a beer
August 11th, 2007 at 5:47 PM
I can’t log in or update, im having to do it manually and still will not work. Is this causing the problem on my end, or am I just having a problem for the first time in 10 months?
August 11th, 2007 at 5:48 PM
Good work catching it. As a network egineer I can appreciate what you have been through…been there many times. Thanks for letting us know a few details, it helps many of your members understand what is happenig and enables them to see you are working on the issues.
August 11th, 2007 at 5:50 PM
And where is the Linux viewer for Voice? I see that Windows and Mac have avai;lable updates.
August 11th, 2007 at 5:51 PM
Yes, I agree thyings do go wrong. Its a very complicated piece of software and hardware thats running the grid. The problem is none of the problems we have experienced this week are new, they have been ongoing for at least since i started SL 13 months ago.
We can live with delays in additional features being added to SL. The problem is adding more bricks to a already shaky foundation will, at some point, bring the whole structure crashing down on all our heads.
Get SL stable. Get it running as it should, then and only then add new eye candy.
August 11th, 2007 at 5:52 PM
Thank you for explaining what has been going on. I realise a technical explanation goes over many people’s heads, but you do need to give details i.e. start with the Executive Summary and then give the real reason. Yesterday’s explanation of a “RAID failure” didn’t cut it with me. Today’s is much better.
August 11th, 2007 at 5:52 PM
One week… only one single week without any kind of log in problems, inventory loss, rezzing failures, grid downtime…*sighs* Wouldn’t that be nice…
August 11th, 2007 at 5:52 PM
“The end result is a pretty rough week for Second Life Residents, and I can only offer apologies for that. The good news is that we have many projects in the works which will help improve the reliability of SL. We’re working on new infrastructure which will help us withstand the failures above, as well as store and transmit data more reliably, and we’re working on reducing both client and server crashes. Several of these projects are new, thanks to our growing technical team. We’ll share more information on these efforts in the near future.”
in other words, LL is starting to listen. really…. we can go without voice or fancy skies, but what we cant go without is a reliable and fast SL.
August 11th, 2007 at 5:56 PM
QUOTE: This is apparently due to a bug in the router model we’re using, which the vendor believed was fixed but which, it seems, wasn’t really fixed.
Bummer aint it LL when people tell you they’ve fixed something and it just gets worse.
August 11th, 2007 at 5:59 PM
PS: My sympathies on the VPN tunnel problem. I have to use them occasionally when telecommuting and they can be erratic and hard to keep open.
August 11th, 2007 at 5:59 PM
Over 2000 servers, in 2 cities. Im surprised SL works at all. Thanks Lindens, for keeping us inworld!
August 11th, 2007 at 6:03 PM
Ian,
Thank you for letting us know what was going on and why. It doesn’t undo any of the problems it caused the user community, but at least it’s a step in the right direction by keeping is informed.
Now we have some idea why things have been the way they were over the last 4 (more like 5) days.
I hope that LL continues to be this informative in the future. It will go a long way in letting the users know that we’re not being treated like mushrooms - kept in the dark and fed a lot of bullstuff.
August 11th, 2007 at 6:04 PM
As a former computer tech (hardware, discrete component level), I fully empathize and sympathize. Simultaneous issues are a bear to troubleshoot. Given what your crew were up against, I am amazed at how quickly and well you kept things running. Well done to all!!!!!
August 11th, 2007 at 6:06 PM
It’s frustrating that we are having all these problems, but it makes it MUCH MUCH easier to tolerate when LL actually gives us updates and explanations, instead of just acting like it’s not happening. This post is exactly the kind of information we like (need, deserve, whatever) when there are problems like these. Keep it up and thanks.
August 11th, 2007 at 6:06 PM
So . . . when is all this mess going to be fixed?
Do we have to wait for the “several projects in the works?”
(And Voice does nothing for me but give me YUGE packet loss. I had to go back to the previous version in order to move in SL at all. Shouldn’t that tell you something?)
coco
August 11th, 2007 at 6:07 PM
i’m laughing #27
August 11th, 2007 at 6:09 PM
Wouldn’t it make more sense with a mission critical application to not rely on the internet for transport? IPSec adds packet over head, and the internet is prone to slow down when big things happen.
If I was looking for high speed bandwidth between co-locations i would be looking at multiple MPLS pipes. You can get gigabit speeds easily, and the cost is lower than frame relay on a per K basis. The other option would be to rent some of the dark fiber that is in the gorund and run direct.
There is, with VPN any way to ensure capacity, reliabiltiy or QoS of your connections.
August 11th, 2007 at 6:10 PM
Thanks so much for the update! Nice to hear why things have been bumpy lately and that fixes are in the works.
August 11th, 2007 at 6:12 PM
@Tom Bender Why should we stop whining here? We pay for second life and demand service all we get is rubbish and be kept in the dark of what is going on!
Maybe you should stop giving compliments here to people that don’t deserve it! Most subscriptions i have envolve much less money, yet the service is 10 times higher!
August 11th, 2007 at 6:15 PM
Good to know, and thanks for being honest and upfront about the problems many of us are currently experiencing. I can only hope that these issues will be a thing of the past ; in the very near future. I am glad you were upfront; as these issues have wavered my support of Second Life in recent months.
Catherine Cotton aka Martin Magpie.
August 11th, 2007 at 6:18 PM
Thanks for the details. I don’t understand a lick of it, but I’m sure you have someone that does. What I *do* get kind of explains why the animations in my eCollar seemed to suddenly stop working, though I had not moved from one sim to another.
Keep cracking! I’d send donuts if I could.
August 11th, 2007 at 6:18 PM
Thank you for the insight, however i belive next time it might be better to explain it as its happening than after >_> many of us begun to suspect things
August 11th, 2007 at 6:21 PM
I really appreciate the details about the problems and hope this will be a continuing trend. Communication seems to be lacking lately in these problems.
August 11th, 2007 at 6:23 PM
Ian, thank you very much for a thorough and informative update. It really helps put things in perspective, and I appreciate all the effort you and the rest of the team are putting in this weekend to deal with these myriad problems. Good luck!
August 11th, 2007 at 6:23 PM
I barely understood the explanation, but it was nice to finally hear whats going on.
August 11th, 2007 at 6:24 PM
Kept in the dark? How does posting a full run-down of this week’s problems and a plan for the near future equate to keeping us in the dark? I think that LL provides plenty of info about SL’s growing pains and problems, more than most companies would be willing to share.
Thanks for the status update, LL. Keep up the good work!
August 11th, 2007 at 6:26 PM
I got stuck in a VPN tunnel once during an earthquake. I can tell you that is no place to be during a big 6.6′er! I guess taking BART to SF from the east bay could be worse. Damn VPN tunnels.
August 11th, 2007 at 6:27 PM
It is not 100% clear if it is related or not, but it seems over the course of the crashes my router got fried.
It would have just been nice to have a note, that something goes wrong with logins. Like with the DNS server problems in Germany, that required to use an US DNS server: no note, nothing on the blog! (just some helpful people on the betagrid who new a solution)
If all your friends are obviously online, but you can’t get in, you really think the problem is on your side!
Now, it is good to know at least a bit of what was going on, or still is going on (even if it is difficult to understand). I am still not able to stay longer than 10 seconds in SL without getting frozen. But, then, my router half died, maybe because I was trying to hard to figure out what was going on. (So, SL friends, see you back on Wednesday. When I will hopefully get a replacement)
August 11th, 2007 at 6:28 PM
Thank you Lindens for not just ignoring us and starting to give us some of the reasons why the probelms, it is much appreciated, and we can start to understand some of the problems you are coming up against trying to scale this world of ours for more people to use.
August 11th, 2007 at 6:30 PM
It is certainly working better, and I find the explanation very useful, as I would otherwise put the previous problems down to my own equipment, network or ISP being intermittently flaky. I hate troubleshooting things that are working properly, you can never fix them.
August 11th, 2007 at 6:31 PM
thanks for all the hard work! cinnabons for everybody!!!!
August 11th, 2007 at 6:32 PM
@Tom (are you on a ) Bender…
Like some of us have posted here; we pay a LOT of $ to work in this environment (if that’s what you want to call it (work; failed TP’s, lost inventory, etc)). When I am expected to pay tier on land that NO one can get to to purchase my products, due to LL’s 20% downtime per month now, I and EVERYONE else that pays tier has EVERY right to complain;
To ALL that say we are “whiners”; if you dont like reading the complaints here, dont load the page. THIS IS THE ONLY place LL seems to listen to the moral majority. I would much rather be called a “whiner” than a “whino”
August 11th, 2007 at 6:38 PM
Well… let’s see if all of this is going to lead somewhere…
And agreeing with a lot of folks around here.. get RID of the crappy voice feature… Skype is so much easier…
While your at it… just make one thing for sure… get that asset server working like it’s supposed to pls pls pls… I still don’t understand why this has to be a problem all the time… MySQL is SOOO easy to use LOL…
anyway… hope next weekend will be better *grins*
August 11th, 2007 at 6:41 PM
While the explaination is nice, my main concern is the money I’ve lost due to inventory loss. I doubt that SL will offer compensation, but someone might want to warn people faster in the event that it happens again. The fact that for several days I was in the dark about why I wasn’t getting my inventory items rezzed or even returned is a pill I don’t want to swallow. The fact that I am only left to ask for items back (some of which have still not returned) makes me even more bitter. When SL stops having it’s continual problems is when I’ll start paying for an account.
August 11th, 2007 at 6:41 PM
thanks for the details.
August 11th, 2007 at 6:42 PM
I’m hoping Ian’s post explains why some of my house is missing… walls, floors, whole rooms. This has never happened to me before in nearly a year of being in this house. Funny thing is though, that although I can’t see the floor in one place (just a drop to the ground below), I can walk on it as if I’m walking on air. Has anybody else experienced similar problems?
I hope your fixing all the problems sorts this out too, it seems as if it will.
August 11th, 2007 at 6:48 PM
why even post a comment. they are ignored. the same symptoms of a cobbled together architecture are ever present. despite a huge objection to a certain feature it was shoved in and then the system goes brain dead for a week+ resulting in income losses for the residents.
why even bother trying to be transparent if you cannot bring yourself to be transparent enough in real time to just tell everyone at logon that they cannot spend or build without major risk of loss?
there isn’t much confidence left for linden research now. it will take a very long time to repair this sort of reputation damage.
try listening to your customers instead of sneering at them.
August 11th, 2007 at 6:49 PM
sorry for all the problems, but why is it when you get one thing fixed everythign else goes haywire, gawd if this was RL ppl would have already taken their business elsewhere, might i again suggest a suggestion box( s) in sl so that those of us who do play in world can offer suggestions ( not stupid shit either) that might just make it a pleasant experience for all
* offers a toast* ” here is hoping for a better SL week next week!”
August 11th, 2007 at 6:51 PM
Hi
I work Worldwide Customer Care at AT&T for clients like SL. And yes we do have Service level issues from time to time. I don’t know if SL is our customer (I’ll look). But as a provider I acknowledge that providing this service to our customers is our highest priority. And when problems arise we will do everything in our power to resolve them.
As a provider, I want everyone to know that we are concerned about how these issues affect all of you. Its not the revenue that counts. Its the benefits our customers enjoy. And a guarantee of that enjoyment.
For me personally, I’ll take this one to heart. And I’ll make sure all of our customers get the service we promise to deliver to them. And thats you. Its a tall order. But an order that we agreed to fill.
August 11th, 2007 at 6:53 PM
Once again we have fundamental problems with this game and the users are divided into two factions: the blind Linden supporters, who seem to believe that LL can do no wrong, and the people yelling at LL to fix the problems before introducing new features and new bugs.
Except you guys need to get something through your heads: The people who work at Linden Labs aren’t listening to you tell them what to do. They don’t care how much you pay. I knew someone who spent $2000 a month on this game. $24000 a year. When he asked LL for help, they shot him down and blamed his problem (a badly crashing sim that needed to be reset every day) on his own doing without even bothering to explore the idea that it might be a problem on their end.
It’s a known issue that the money we spend goes to whatever they feel like working on–not what will improve the experience for its paying customers. Unless there is some dire emergency that requires intervention or the game will end up not working at all, they feel your money is much better spent on development rather than quality of the product. It’s time to accept that and either deal with the fact that the Linden staff have no souls or concept of customer service, or stop using the game (or at least stop spending money on it, like I did–Linden Labs will not get another penny of my money until they give me something worth paying for).
I can only hope that Linden Labs will see the errors inherent in their business practices someday and will look at their game, which will by then not even have five-digit concurrency any more, and realize they made a mistake.
I hope none of you are holding your breaths for that, though. It could take a long time.
August 11th, 2007 at 7:06 PM
Thanks for the heads up. It has indeed been a rough week and I want to thank you for taking the time to explain what happened. I don’t understand most of it, but I get the gist.
And I agree with some of the other comments - fix the current problems, make the grid stable, THEN go for the voice and pretty skies.
August 11th, 2007 at 7:16 PM
I, for one, welcome our tunnel masters to the 100th level of experience.
Lindens, Thank you. A few forthright details can make a big problem easier to swallow. Thank you for those details.
August 11th, 2007 at 7:16 PM
Again Dallas? Thank you for clearing reason. Its helps understand alittle more then if not explained.
But are we going to get replied in anyway?
August 11th, 2007 at 7:17 PM
“Thank you Lindens for not just ignoring us ….”
LMAO ! I hit the floor on that one..
August 11th, 2007 at 7:17 PM
We need some more solid and tangible reassurances here.
I am relatively new to SL, and it feels like this sort of thing is all I know of it.
Never the less, I have been spending 10 to 18 hours a day trying to set up an business, and considerable cash, and I have to say it feels a bit like throwing it into the wind and just hoping that it finds its way.
This is big money now, a huge time and investment for so many - peoples livelyhood - and you can no longer afford the luxury of feeling that you are “allowing’ us to play in your sandbox.
The fact that we have absolutely no way of backing up our work done online to hard disk is a giant leap of faith for us, and for what you charge us for the privilege of doing business with you, you HAVE to offer us some security in that.
To loose a shirt, or a jet ski from my inventory that I paid the equivalent of 25 cents real money for - well, I’ll get over it, but to see that happen and then think the same could happen to all of my inventory, my time, money and hard work…. well, how would you feel?
August 11th, 2007 at 7:25 PM
Thanks for the information. Yes indeed a ROUGH week lol. But I must also offer my 2 cents…..
The Trevally Sim is horrible. My objects fail to rez, or when rezed sink into the virtual earth as though they are sinking. Secondly, objects in BUILD MODE do not behave correctly, ie becoming unlinked spontainiously, will not accept position commands, will not accept delete commands, can not be taken into inventory, etc etc.
Personaly I believe voice has over taxed and crashed your hardware. And on the subject of VOICE, I have yet to meet a resident, paying or not, that welcomes voice into our second life. Honestly it seems as though Linden Labs proceeds with ideas without any regard for what thier customers actually want in thier second life. Voice is for FIRST life NOT second life.
Thanks and happy grid-running!
August 11th, 2007 at 7:27 PM
WOW finally someone who tells us more then we appoligise and thankyou for your patience. Kudos IAN for actually taking the time to report to us on what is /has/was happening. We all know you don’t/can’t control how hardware and your service suppliers work. My hats off to you Ian for Doing this maybe you can rub off on the rest. But yes i agree fix current then consider other options
August 11th, 2007 at 7:33 PM
Not ignoring us? they’ve been ignoring the problems & us since I started in SL tho I have to admit it worked better then than now. The letter from the residents asking LL to repair what needs fixing & stop adding stuff til it is, wasn’t worth the paper it was written on. Basically LL said ” Ok, we got the lettter now go play your game & give us your money.”
Ok, so we know what the problem is. Is this fix just fixing the symtoms or is it going to fix the base of the problems in SL? I’m tired of not being able to work & build in SL because of one problem or another.
August 11th, 2007 at 7:36 PM
“The end result is a pretty rough week for Second Life Residents, and I can only offer apologies for that. The good news is that we have many projects in the works which will help improve the reliability of SL. We’re working on new infrastructure which will help us withstand the failures above, as well as store and transmit data more reliably, and we’re working on reducing both client and server crashes. Several of these projects are new, thanks to our growing technical team. We’ll share more information on these efforts in the near future.”
Well lets hope this is true. But when I see it, then I will believe it! How many times have we heard the ” it will never happen again”? I Just hope you people learned Outdated hardware needs replacing PERIOD! What I don`t see really in this meassge is besides the HDD being replaced, not nothing else is? So inturn i seeing a repeat of failures based of the lack of better judgement of replacing old and warnout hardware.If we were sitting at 2 million accounts, I say ok you use current hardware. But we are not reaching 10,000,000 accounts spells ! MELTDOWn. If hardware is not replaced because of the heavy useage. So Ian, if you really want us the users to be relaxed do a little pervention. REPLACE PARTS THATS NEED IT!
August 11th, 2007 at 7:38 PM
FINALLY a real explaination. I think many SL users get more upset at some of the generic explainations than the actual crashes. NO one wants crashes, but treating paying users as slightly dim kindergarten students only makes matters worse. Thanks for the explainations, good luck with the repairs, and don’t ask for a downtime refund from your service providers unless you offer one to us as well.
August 11th, 2007 at 7:40 PM
Since my other post was moderated I suppose I better keep this “fan boy friendly” and recommend another slice of last weeks compensation be given for the repurchase of lost inventory or whatever way LL can provide its “paying” customers.
August 11th, 2007 at 7:43 PM
Wow.. Tough week.
/me buys the grid monkeys a case of banana bear.
August 11th, 2007 at 7:43 PM
THANK YOU. A week or two ago, I asked that LL provide it’s residents with as much info as possible regarding problems as they are encountered as well as keeping us regularly updated as to the status of the problem. Maybe you saw my post, maybe you didn’t, but it’s good to see this flow of information coming out of LL. Knowing what’s going on makes dealing with it that much easier.
Cheers, and thank you for keeping us up to date.
August 11th, 2007 at 7:44 PM
Hi,
I really feel left out when everyone around me voices.
On the table in front of me is a brand new headset, and my PC even has a connector it fits into - but I am on LINUX and you keep delaying the Linux voice release and even refuse to give an ESTIMATED time.
Please let us know if we can expect voice in a week, 2 week, a month, 2 months or a year…. I mean, really, you must know SOMETHING!
On another note, @26, I actually went and the place is a casino, just by a different name. It is exactly the kind of place that blights SL, with loads of camping and things that take your money.
What a devious move, using the very blog here to advertise a casino!
August 11th, 2007 at 7:49 PM
Heh.. I and my partner appreciate the info. Thanks, Ian. Noticed the oddities when the red scripted object beacons in the 0,0,0 corner of the SIM appeared frozen. Usually, like prims and avatars which appear in those corners, script updates only flash in and out in a fraction of a second. Lately, they have hung there like small red novas. Curiously, they (and all the prims and avatars) appear in that corner in relative positions to each other that they will have when they finally do get placed in world.
August 11th, 2007 at 7:51 PM
Wow wee…
Nice to see everybody back on the blog..it was a rough week..SL crapped out so I actually read a book and checked out some other virtual worlds. Lindens are still ahead of the curve it would seem to this neophyte observer of this kinda thing, but competition exists for those of us suitably po’d.
I’m not…with the exception of this past week I think I have had good value for my small monthly tier and premium account. endless amusement frankly; in particular when somebody’s hair rezzs in their crotch or a shoe comes through the ear.
I feel bad for you folks not getting satisfaction but a great many people do, and having been an SL victim of inventory robbery before($20kL) I sympathize…but in the end you can walk if you want.
It’s the only option, be sorry to see you go but a smaller game would be better generally I think. I don’t like the label whiners because if you pay you have a right to complain and THIS is the place.
I’ve met piles of folks loving voice and awaiting the sky thingy whatever it is, but yes…keep those tunnels open Lindens and shore them up before showing us the new sky. It’s sometimes very dark in the tunnels and the place is full of trolls and faeries and vampires, and goreans everywhere you look..nevermind the furry things scuttling about and getting in my wife’s jam.
August 11th, 2007 at 7:53 PM
It is good to see things explained in depth now and it makes it much easier for us to have patience with Second Life when we know what’s going on and what’s being done about it. I really appreciate this news!
August 11th, 2007 at 7:59 PM
WORK ON THE BASICS! I could care less about casinos and the things that “blight” SL ( some of which I enjoy). if you don’t want to go to a casino and camp then don’t go! I personally think the things that “blight” SL is SL working poorly- less time cleaning up our activities and more time making activities possible_ it was my understanding that SL was for us to create- if it works properly we can do that providing we don’t get stuck with a bunch of well meaning moralists-where did they come from anyway?
August 11th, 2007 at 8:00 PM
I very much appreciate the update on what happened. Most of the details are more technical than I need, but it gives me a feeling that someone cares, and is on top of things.
I am in strong agreement with those here who say that LL should focus on getting things working well, and keeping them working well, rather than on “enhancing” our experience with weather that no one can share and other such new features. I am keeping an open mind about Voice, and can see some places where it may be useful.
HOWEVER, you promised that voice would not affect performance, and everyone I know who has downloaded the voice client is reporting many many crashes, as often as every 10 minutes, even with voice turned off. That’s a performance issue, and a big one.
I have heard that LL employees get to work on whatever they want to work on. That makes it a great job, I’m sure but it doesn’t make my Second Life a better life.
If you need help learning how to do testing and root cause analysis and other modern approaches to improve things, I can put you in touch with the consultants and training people we have used in my company to our great delight. I’m sure they’d love to come help you.
Thanks,
Janet Rossini
August 11th, 2007 at 8:01 PM
[...] On Wednesday, we experienced a failure wherein our primary bandwidth provider started dropping about 95% of our IPSec traffic bound from one site to another, but only for some of the tunnels. Other, non-IPSec, traffic was largely unaffected, and so this was a very difficult problem to diagnose and, crucially, get the provider to fix. Source: More info on this week’s grid problems [...]
August 11th, 2007 at 8:01 PM
well at least I know why I suddenly went from 6ft 9..to 5ft 2 and no eyes and looking like I had mange and..and..hey…it could be a lot worse..there could be no SL.It seems to me that the wonderful times I have in world are well worth the problems that I sometimes experience..(I’ve tried stomping my feet and throwing a tantrum…doesnt work any bettr here than it did when i was !) Thanks LL, all in all we’ve got it made!
August 11th, 2007 at 8:02 PM
Thanks for the explanation. Doesn’t sound like it’s been a fun week for anyone.
August 11th, 2007 at 8:04 PM
@53: I’m not a “moralist” - the blight comes from camping, which ruins entire sims.
Incidentally, if all the connections used by camping Avies were dropped, SL would have 50% less load and would maybe work right.
Think about it. I belive camping has no place in SL.
August 11th, 2007 at 8:05 PM
gripe, gripe, gripe….
complain, complain….
whiiiiiiiiiiiiiiiine!
Just my 2 cents
August 11th, 2007 at 8:08 PM
Don`t sence of the issue here. it was hardware and networking issues . Don`t start taking things off subject matter and bring in campers. We all know the problems they cause but lets keep this issues on topic. Atleast for the sake for the future for it to not letting it reoccure again. Stay focused
August 11th, 2007 at 8:08 PM
thx for the update Ian …. and for those whiners just be happy sl is running
August 11th, 2007 at 8:09 PM
Excuse me, Mr Linden.. when will it be safe to install things on my clit without losing it?
August 11th, 2007 at 8:11 PM
cheese anyone?
August 11th, 2007 at 8:14 PM
O cool - I just spent $1499L on an item I never received - I guess I’ll have to rely on the good intentions of the vendor to make good - even tho it’s not his fault. Or dance for another month to make it all back so I can wait for another bug to screw up my transaction again.
Lost inventory should be compensated - it must all be logged somewhere and easy enough to verify.
This is not the first time its happened this month either - hardly good news for vendors and the SL economy - I’m getting nervous every time I purchase something and that is NOT good for business.
If I ever get my non-delivered item, it’ll be the last big purchase until I can be guaranteed I will get what I pay for - if this were RL and I didn’t get what I paid for, consumer laws would protect me - what is Linden doing to protect consumers and is the lack of protection in breech of US laws - u stopped gambling in SL because it broke RL laws - do the same for consumer protection sometime now-ish!
August 11th, 2007 at 8:20 PM
If second life is to succedd and bring in more investments, as in ‘land purchases’, then it HAS to be more stable. To many people are getting discouraged and turn away from the game. To me that is a shame becasue I , and others, see the potential here. PLEASE PLEASE put all your talent on that…STABILITY!!!!
August 11th, 2007 at 8:23 PM
Good communication LL… it makes a big difference when we know what’s actually happening! Thanks.
August 11th, 2007 at 8:27 PM
Melanie, some of us don’t have credit cards or live in countries that SL will not allow credit use so camping is one of the few legal ways to make money in SL - Your attitude is elitist and patronising. Get a RL and let the campers camp. There’s plenty of other sims with no camping to be by yourself in if that’s what you want.
August 11th, 2007 at 8:29 PM
#30
It would be nice to know why their not doing that.
August 11th, 2007 at 8:33 PM
Ah, it all makes sense now. Thank you LL. Finally we are told what happened. If only you told us this sooner xP. Anyway.. I downloaded the “voice-enabled” client, but not the newer version, simply because I never use voice. Either way, though, You guys are doing quite well at keeping SL going, even with all the problems. I think I’ll Volunteer, so I can help you guys with the issues, and perhaps let the people be heard ^.^ Anyway, I hope all goes well, and everything behaves from now on.
PS: Ignore all coments that complain about the same things. the “No gamble” rule was a good move. those machines did cause lag, and, Voice is cool, but unnessecary in my opinion. I’ve never really crashed, and I’m happy with SL.