Post-Mortem: September 6 web caching issues
Friday, September 7th, 2007 at 4:31 PM by: Everett LindenAt approximately 6.30 am PDT, we noticed a significant concurrency drop. Members of the Operations team identified that there was a serious problem that prevented logins and the ability to fetch assets. Specifically, squid (web caching software) was no longer running on the majority of simulators. The issue was due to a broken file permission. Since all of our HTTP traffic flows through the local squid process, squid not running meant that no asset traffic was running. As a result, auto-save, asset saves and asset loads would not work.
Operations immediately begun a workaround to restore these sims as fast as possible. This resolved a large part of the problem, and assets that had been queued up now preceeded through the proper path without loss. Yet we noticed some failures after the “fix”, such that a small percentage of assets were being lost on certain sims.
At about 9.30 am, we uncovered the actual bug. The operations team fixed the script that starts up and shuts down squid and redeployed the script. This was much more successful, and we had consistent performance with no asset loss. Concurrency began to recover at 8am, and was fully recovered by 11.15 am.
This bug will not recur, and it helped the team to identify opportunities to improve the bug fixing process as part of our ongoing stability efforts.
We apologize for the inconvenience, and we appreciate your patience during the fix.
Have a great weekend!
September 7th, 2007 at 4:40 PM
Well done, guys!
You rock, all of you.
September 7th, 2007 at 4:43 PM
It certainly makes a difference when you explain the problem and the resolution path - in non-geek language).
ta
September 7th, 2007 at 4:45 PM
Thanks you for actually informing us about the details of the situation. It is very much appreciated ^^
September 7th, 2007 at 4:51 PM
File Permission error’s are one of the hardest to identify, and can cause the most hair to be pulled out of one’s head! Kudos to the server folks, Be Proud - You guys and GALS do rock!
September 7th, 2007 at 4:51 PM
you guys and Gals deserve a cookie!! sure i gripe when sumthin aint workin,but when its caught and fixed properly, i give you a handclap
September 7th, 2007 at 4:52 PM
I used to agree with those who criticized LL for focusing on new features rather than fixing bugs, but this shows that LL’s bugfighters are alive and well. Your job’s not at all easy. Thank you!
September 7th, 2007 at 4:54 PM
Am I the only one who is worried about this? There must be a lack of change control for file permissions to suddenly be wrong on the main grid.
More control, LL, things shouldn’t “just change”.
September 7th, 2007 at 4:59 PM
I have many years’ experience developing software and creating large, scalable Internet services at one of the world’s biggest software companies. I just want ot say that I think it shows an incredible ignorance of the prevailiing standards in the software-development industry to say “You rock.”
The fact is, a data-loss bug made it into the production environment. Furthermore, this was not a rarity but it’s a norm. Normally business that charge customers money for their service invest more to deliver quality service. In particular, a test team of comparable size to the product development team would normally maintain a parallel grid with simulation of load similar to that in the live grid, including some worst-case or projected-future loads. Changes to the SL server enviroment would be thoroughly tested in that parallel universe before customers’ assets were put at risk.
If the test team were unable to keep up, new features would be slowed down and/or business policies would be changed to ensure stability for paying customers (temporarily suspending creation of non-paying accounts being the most obvious such option).
This doesn’t appear to be the Linden Labs way. Perhaps one day they will find the religion, or perhaps one day an able competitor will come along to demonstrate to LL that such a mode of conduct makes for happier customers and, in the longer run, a more successful business.
While the efforts of many individuals at LL are laudable, the priorities and the allocation of resources coming from the top are flawed. We deserve better.
September 7th, 2007 at 5:19 PM
great work, omg i was begining to worry, exelent job, there’s now “true” tp ussues and you guys have been on fire with help and all, wtg *4 thumbs up* lol
September 7th, 2007 at 5:28 PM
this is the best blog post ever
September 7th, 2007 at 5:31 PM
How refreshing to see an honest report about something that went wrong, an explanation of how it was handled, and the resolution of the matter… and COMMENTS allowed! See, that wasn’t so bad, was it?
September 7th, 2007 at 5:33 PM
OMG, the grid is so screwed up still. Sims crashing, TPs not working. Glad you think its all resolved, typical.
September 7th, 2007 at 5:34 PM
Not all, but this one.
I like the ones where a definite cause can be found. You fix it and it stays fixed.
WTG
September 7th, 2007 at 5:34 PM
i agree with Demelza . What just some random linden playing with buttons they shouldnt? the thing was fine before. then its messed up.
sighs…. and for people that seen my last post on another issue saying “why do i play when i hate sl” you all shut up. i didnt say i hate it i said its poorly maintain.
Me I love SL. i just hate how its poorly maintain. Look at the there blogs there a bug one day then the next day. then a rare of a day gone passed without reported issue then there another the next day. alwas log in issues and asset issue. clearly there something they gotta stablise.
I seen dial up have better up time even with people picking up the phone and noticeing someone online and hangs up. and it staying connect.
i so hope they stop with the adding of features and work on perfomances and bug at hand. stablise the cilent and grid.
September 7th, 2007 at 5:35 PM
This is good service
September 7th, 2007 at 5:36 PM
p.s and Melanie . that fix it, it stays fix dont apply to linden labs. too many of the same fixes breaking again and again day after day. lol
September 7th, 2007 at 5:42 PM
And yet, sadly, I still can’t login in. Fixed don’t mean fixed I guess.
September 7th, 2007 at 5:50 PM
I, personally, like the post-mortem/after action review of this issue. It shows that someone actually wants to communicate with us residents instead of treating us like mushrooms.
Unfortunantly, some people aren’t ever out of “Gripe” mode.
September 7th, 2007 at 5:52 PM
Can’t say that I was affected by the outtage, but that was a pretty impressive explanation
September 7th, 2007 at 5:55 PM
Things just don’t happen…things happen when you mess with something and oops you break it. If its not broken don’t mess with it. Focus on what you have first. You LL folk are the like silly children playing with blocks…you stack on block ontop of another. you start to notice is starting to lean to the right alittle bit. Any Reasonable person would try to stablize the tower before you build any higher. Do you Do this LL? No you ignore the fact that is unstable and keep building up until it falls down. SecondLife and a Tower of Blocks are not much different if you don’t stablize the system its going to blow up in your face.
September 7th, 2007 at 5:59 PM
Um…….Sounds like they fixed it. Lets hope it does not reoccur. Thank you Guys
September 7th, 2007 at 6:01 PM
I totally agree with you Troy….I don’t want new features, I want the standard stuff to work…..so glad I was napping through this most recent episode, I’d hate to loose a few thousand L$ more worth of inventory.
September 7th, 2007 at 6:13 PM
After LL proclaimed an resounding [RESOLVED] on this issue… it still took me over an hour to log-in. I’m glad LL can reach their own back for a pat.
September 7th, 2007 at 6:17 PM
I have to agree with Post #2 by shai Khalifa. It makes a very big difference to those of us who enjoy SL, but are not necessarily familiar with various types of hardware, software, and various bits of computer jargon. I am lucky–I have a lot of help and support from computer savvy friends (RL and SL), and I am learning to identify what bugs are coming from my computer vs. bugs/ongoing issues from LL, along with programming and building.
I do give you kudos for this post, and for allowing people to submit their comments. However, I sincerely hope that you do take more notice of from those of us who put a great deal of our own money to fund projects in SL that are not making us money, but that we simply do out of a desire to create, explore, and, as I always hope, have the freedom to make our own societal mistakes and learn how to rectify them.
However, we need LL to become *much* more supportive to those customers who may not necessarily be monied corporations. Please remember–your base will always be those of us who continue to use SL not as a money maker (although, many of us hope to at least break even!), but as a way to build and sustain supportive communities. You need to pay attention to those of us (the common masses?) who lose money, items, and assets due to problems on your end.
LL, no one likes to be yelled at continually, especially if you are in a situation where you are understaffed or are having other internal difficulties. However, being direct and honest with the citizens of SL, as you have done in this post, in a clear, concise, and direct manner, will help you even more.
In addition:
If you want the citizens of SL to respect you, you need to grant us the same amount of respect, even if we are not those who own a sim, a large business, or other profitable venture. I am *not* saying that those who have made SL-based profitable businesses, or those who *do* make a living from SL are bad, evil, naughty, or selfish. I think most business owners are committed to their businesses and their customers, and the support I have gotten from business owners in the past surpasses the support I have often received from LL.
Please stop hiding behind a TOS agreement, and be big enough to help those who have lost money and assets due to problems that originate in Linden Labs. You will be amazed at the amount of customer loyalty and support you will get when the corporations who are now moving into second life disappear because there is no one around to purchase their products or use their services.
September 7th, 2007 at 6:32 PM
Let’s hope Cyber Terrorists don’t get any (not so) wise ideas on the 11th
September 7th, 2007 at 6:54 PM
Seem to be MAJOR problems again in-world tonight with lag and packet loss but no announcement here, no noices in world and still no way for us mere mortals (basic account users) to flag things like this up… JIRA’s a great idea for bugs etc but how aboutthe service issues? Notices in-world would sure help as soon as you notice…
September 7th, 2007 at 7:08 PM
LL needs to do more post-mortems like this. Posts like this are well-worded and explain exactly what went wrong.
September 7th, 2007 at 7:11 PM
Good job guys. All issues should be taken on in this manner.
September 7th, 2007 at 7:15 PM
when the asset sever is in TROUBLE as soon as u know you need to WARN PEOPLE right away with an IN WORLD MESSAGE WE ARE SICK OF LOSING STUFF GET THE PICTURE YET?????????? good u fixed it but bad for all of us that continue to lose so much money in lost inventory cus NO ONE WARNS til hours after the fact… i dont know if you realize how much you hurt in world business with all these inventory losses WHO WILL WANT TO PURCHASE anything in game when its 99% chance it will just poof with out any recourse what so ever. I think its high time u all worked on this cus nothing has been done to resolve this bug yet we should NOT be losing things as much as we are we need a BETTER way to back up our stuff if you cant do it for us
September 7th, 2007 at 7:20 PM
Thank you very much LL.
Simple, straightforward, and to the point.
Every resident knows that LL hates bugs as much as we do — thank you for your continuting efforts to tell residents what happened and why. This blog post is exactly what residents are asking for…
September 7th, 2007 at 7:43 PM
I hate permissions errors. THey are the bane of my existence in Linux. Glad you guys knocked this out quick.
As for the whiner in message 19 above me, maybe they COULDN’T send some sort of warning because the system for doing so might have been affected by this issue. THINK before you type, PLEASE.
September 7th, 2007 at 8:07 PM
I’m just adding my compliments to LL. This occurrence didn’t affect me this time but I appreciate the post and the fast fix. It’s probably a good thing this happened on a Friday rather than Saturday or Sunday.
September 7th, 2007 at 8:12 PM
Yeah but…
If you were rolling out an updated script that loaded squid (a vital part of every sim). Or even just installed the latest version of squid. Would you perhaps test it in a safe off grid environment first?
And if it’s not directly squid related. If you’re rolling out some other patch that affects file permissions. Wouldn’t you want to test that off grid first?
Am glad you guys worked so hard to fix it. And thanks for that! But … ?
September 7th, 2007 at 8:13 PM
My complete sympathy, I chased one of those for almost a day right at a quarter end. What a pain in the a**. Hang in there guys.
September 7th, 2007 at 8:16 PM
@ 19 Go here and say that. Because many of us are wondering what is LL thinking these days? Look at al the Lovers of LL and its strange and at times off the wall the wall Linden posting there. At this pace how long does SL have if the present conditions are not improving.
September 7th, 2007 at 8:18 PM
STOP FIXING BUGS AND ADD NEW TOYS..
Oh.. wait, I mean.. erm.. actually nevermind.
Good job fixing it. Might have been good to drop those of us inworld a blue-note to let us know there were problems, and to “be careful” with no copy items.
September 7th, 2007 at 8:19 PM
@ 20 19 does have a few points. People don`t like losing items etc.
September 7th, 2007 at 8:20 PM
Good info on the fix and the issue. Thanks for that. Now how about for the next month nothing new…maybe even no new land…have everyone at LL focus on bug fixes and grid stabilization and excellent customer service. Give us a way to back up our own inventory. Thanks for the improvements many have mentioned so far…good work!
September 7th, 2007 at 8:23 PM
Oh, and number 20…stop insulting people who blog. Who do you think you are anyway??
September 7th, 2007 at 8:30 PM
I love how the team didn’t call it “good enough” and go back to their card game. They stuck with it and looked for the real problem behind the apparent problem.
Then, as Evert Linden pointed out, they used this SL problem to identify opportunities to improve the bug fixing process. This is the kind of thinking that hasn’t been going on for a long time. WOOT!!!
September 7th, 2007 at 8:55 PM
Well done, LL - This is an improvement in a number of ways.
Not accepting the obvious problem and going on to find the real one.
Great.
Using a post-mortem entry to the blog to answer ‘what when where why’ of things.
Great. It gives us some understanding of what’s going on - and any information is always better than none.
I’m not in a position to judge how true the ‘this shouldn’t have happened’ points are, above. I see a number of folks who work with these types of issues have given you sympathy, so I’ll assume that this kind of thing both ‘happens’ and is a devil to deal with.
Again, great.
I have to agree, however, that alerting people in-world of a majot issue affecting asset loss should have also been a priority. Asset losses have been going on for a very very long time, and I hear more and more people upset about that. Many non-copy items are either expensive, irriplacable or both. Warn us, please, when they are threatened.
Having comments enabled - Thank you. Again, it lets the community respond, positively and negatively - but at least we can see we’re not alone and in the dark about these issues.
All in all, I consider this progress in how an incident was handled.
That said, there still remain other issues:
I’ve posted this before I will keep posting it when I’m able.
Fix The World.
Dear LL;
I’d be a lot more sympathetic to your problems and ‘thank you for your patience’ messages if you hadn’t made this situation yourselves over the years.
The fact is, SL isn’t going to get better quickly, and it won’t get better *AT ALL* if LL doesn’t sit down and fix the basics, as the community has been begging them to do for years. We could have waited for the bells and whistles, and the ice cream on the cake, We understood that SL was to be a work in progress, constantly evolving.
Why didn’t you?
It took a long time for me to move from ’slightly annoyed’ to ‘Furious’ and it will take time for this paying customer to get back to ’satisfied with my SL Experience.’ -
Show me you’re listening.
Show me you’ve changed.
Show me you care.
Show me I’m doing the right thing by standing by you, SL.
Show me I’m not waisting my time.
Fix The World.
To some of my detractors - especially new-timers and those who are non-paying, or think I am ‘unpatriotic’ regarding SL:
‘The World’ was a very different place when I joined in September 2004. I’ve literally ‘paid my dues’ every month since then to LL, and I’ve seen ‘The World’ ’s functionality peak and then degrade while frivolous bells and whistles served as bread and circuses to keep the masses distracted and amused.
(And to court business interests to set up shop in SL.)
I post here because it’s the Official Linden Blog - a main page, not buried in a forum somewhere. LL hasn’t listened to the community in those venues. Perhaps they’ll listen when it’s where businesses considering SL can see it.
I don’t wish to play the ‘Vet’ card, but I’ve been here long enough to have seen. You haven’t.
I don’t wish to play the ‘$$’ card, but I’ve paid for my usage - you haven’t.
The ‘SL Luv it or Leave it’ , ‘My SL right or wrong’ and ‘Post where ur supposed to’ reactions are platitudes and should be on virtual bumper stickers.
I’m not a Joiner. I’m not a Protester. I don’t get involved in Movements and Causes. But even myopic, self involved, apathetic people will speak out when they’ve had enough.
I have.
Have you?
September 7th, 2007 at 9:02 PM
Hey! Where’s Ann O’toole? I need this post explained. A squid is a very intelligent animal but I didn’t think they were that wired a species.
Still and all a good post everet, I got a good picture of what went down, and if you found some new way to fix bugs then it would make life a lot easier. Go after the can’t rez, can’t save, time out on script saves- type of bugs that seem to have plagued my sl this past week at an alarming rate.
September 7th, 2007 at 9:04 PM
Now you can start working on the issues between you and all of SLExchange’s boxes and terminals, don’t you think. Also with the issues of info in “about Land”. Would be nice to have a game that functioned.
September 7th, 2007 at 9:04 PM
@30 well can you sum up in less then 300 words?
September 7th, 2007 at 9:25 PM
hey i had my share of issues with the performance, but if your *example (explorer.exe) wasn’t told to run, there wouldn’t be folders or desktop… lol “squid” didn’t need to fail, but it did, this could of been a string calling the same thing patern done by a tired crew, who knows, well, the sims what where eating me are fixed, ty LL and good luck…
September 7th, 2007 at 9:58 PM
I don’t know what u did to “fix” things, but starting this afternoon(SLT) I started to crash hard again(I had to turn power off, manually). It seem that your server has started again to send erroneous data to the viewer, which causes the viewer to crash. Can somebody look into that?
September 7th, 2007 at 10:01 PM
like sweet (19), i also think it is bad for the economy…strange, the most expensive objects seem to disappear…i dont understand this order.
i also dont understand why the repeated bugs continue off and on…i think a non tech explanation would help many of us…
i love sl…but it is a source of frustrations many days…
i wonder if there is not competition, causing these bugs, giving LL a bad rap, with the intention of surpassing LL in the future of our virtual world web…
if so, we need to be told, so we can understand, and perhaps be of help…we stand to gain if LL merges with top web companies…we are pioneers in this.
the griefer problems need to be resolved, the greedy get greedier…we can change this with cooperation and knowledge to truths.
i suggest also every employee be investigated, any avenue in which someone is able to effect LL…
i suggest scripting be monitored as well, with needed permission before release…this will be hard on the scripters somewhat…but will insure bad scripts are not given to our world here…
sadly as humans, trust has been breached too often, so we must be aware, and secure our tranquility…
‘Together’…the honest can become strong enough to rid virtual worlds of criminal behavors. so let the honest ’stand up and be counted’…and the fakes brought to light…i have a guillitine i bought in world…be glad to use…lol…sorry, sick sense of humor in this…(btw, if this disappears, i am not buy another!!!)
September 7th, 2007 at 10:03 PM
Just a few moments ago some island did a hard crash. what in the world is going on?
September 7th, 2007 at 10:05 PM
dunno but it’s 10:02 pm SL time and I can’t upload textures..I get this error:
“Unable to upload 1fqwy89qw-qw756 due to the following reason:
The server is experiencing unexpected difficulties. Please try again later”
Please fix it b4 you take off for the weekend.
September 7th, 2007 at 10:09 PM
@ (36) Very good point with the griefer issue… i recomended LL try to get the IP Address in one strain of attacks based on that “telltail user patern”… (4 days of new throwaway accounts) that’s a grief no one has to see… (xxx porn, racism, particle, ecs.) like come on, Sony bans IP addresses first… unreal, and i see the point of harsh disepline in that since
September 7th, 2007 at 10:09 PM
oh boy! here we go again!
September 7th, 2007 at 10:15 PM
39 says….”Sony bans IP addresses first… unreal, and i see the point of harsh disepline in that since” We are far from being Sony and LL has no intention to do this type of banning. Why? Because numers of users are more important then a few griefters.
September 7th, 2007 at 10:15 PM
I am sorry but you missed something. Since thursday there has been a decrease in sim performance. You really notice it if you sail a lot in game like I do. For the last two days it has been real bad. Sim crossings sometimes mean you go sailing into oblivion under the world thousands of meters or actross several sims till it catches up, if it catches up. Other wise you just crash. This is not normal sim performance. There has also been an increase in packet loss and freeze-ups that don’t registor on the statis bars. Script performance has been doing strange things also. At times of serious script dialations, I have checked the top scripts and seen items that should never be on the list, at the top of the list and through the roof. Items with just a simple partical effect and stiff like that. Also the boat at times would just stop after a sim crossing and I would have to reset it. This is stuff that has never occured in the past. I think it points to script handlers or something like that. But the point is that this latest episode is not over and is not fixed. You must know this, you must have some way to gauge performance. What has changed between Wednesday and Thursday?
September 7th, 2007 at 10:29 PM
For those that whine about losing No copy Items. I made a few simple prims for myself that I test the waters with before I buy anything. I have not lost anything in a long while. So if you are still losing stuff shame on you.
September 7th, 2007 at 11:33 PM
My computer setup:
Intel Core2 Duo 2.4
2 Gb Ram @ 800Mhz
ASUS P5B MB 1066 UFB
EVGA GeForce 7300 GT 512MB PCIe w/DVI/HDTV Vid Card
CRASH RECORD:
9-1-2007 7 times
9-2-2007 10 times
9-3-2007 5 times
9-4-2007 8 times
9-5-2007 6 times
9-6-2007 8 times
9-7-2007 9 times
I am online quite abit more than others as I am a builder, retailer and buy and sell property. SIM performance is terrible where ever I am at. jerking, stalling, freezing and then there is this crash situation. I do not run a cheap machine but since 2 updates ago everything has gone to the dogs. I get the standard response from LL support - autoresponse - nothing futher to investigate what is happening. I clear cache regularly, that doesn’t help. The crashes come from a variety of actions, doing something in inventory, dropping a prim and just turning around. The only thing that never breaks down is the membership billing server.I understand this is a complicated code for this game. Can somebody please tell me what I can do besides quit the game to improve this crash, jerking and freezing problem. Is anybody else having these problems?
September 8th, 2007 at 12:00 AM
Teleports still borked
September 8th, 2007 at 12:13 AM
hHmm as much as i am happy this bug is fixed.. i guess SL still has a long way to go. Since 1, 2 weeks, the lag is unbearable!!!!!
people in both my shop and clubs are hardly able to walk at peak times, when sitting on a poseball it takes about 10 seconds for it to work.. teleports crash like none before, crashes happen randomly.
So this bug may be fixed, but the game runs worse than ever!! Please g on fixing the bugs!!!!
September 8th, 2007 at 12:14 AM
to all those ’specialist’ never having set foot in the world of servers (especially unix or linux): File permission errors can be the nastiest thing to track down. Ask any system-admin about them and the first thing you will hear is a deep sigh!
I am by far someone who creeps up LL’s behind but in this case they did a good job.
@36
Do you have any idea what your proposal regarding obtaining permissions to upload/embody scripts would require? Any change even a user could make to a script - and be it to add a floating text or item - would require such a permission as there is no way of distinguishing between a minor/good vs major/bad (or any combination of such) script automatically. Good Lord…it sounds so much like the EU commission in Brussels…..
Why certain bugs re-appear after they have been ‘fixed’ is often quite simple: Because it was not the actual ‘bug’ that was fixed but a bug that was caused by the ‘REAL bug’.
I’ll try to explain it in easy words: When looking for the cause of a problem (the things we notice as users as well) one stumbles upon the code error that triggers this unwanted event - we shall name this bug the ‘child-bug’ in this explanation. Unfortunately this ‘visible’ child-bug was the result of another error in the code at a totally different location that ONLY happens in a certain scenario (a very unique combination of factors that must happen for it to be triggered). Let’s call this error the ‘master-bug’.
If this ‘master-bug’ remains undetected and in a very complex setup such as this program it unfortunately is very likely to remain undetected when doing the ‘rush in’ first aid routine it will/may eventually trigger a new or very similar ‘child-bug’ when the triggering event happens again.
SL requires heavy interaction of various parts such as different servers, programs, routines, viewers for it to work. Permissions are not always static (yes/no) but often have to be dynamic (yes if/no if/else….)so unless one is able to isolate the ‘master-bug’ it may(or may not) hit again.
In short - one is curing the symptoms but not the cause!
Griefers unfortunately will always be a thing we will have to deal with at one level or another. There is no easy approach to stop them so we will have to learn to live with it in an open environment such as SL. As little as we can stop crime entirely in RL it will be possible to do such in SL.
One thing that really is noticeable is the fact you mentioned about more valuable items disappearing first (Xcite is the classic example). For that I would really like to hear an explanation from LL at any given time as otherwise it leaves the door wide open for the wildest speculations or is it that we just notice those items just because they ARE so expensive - I don’t know.
UM, are you trying to set a new record in posting to the blogs, so others can not do so, or how shall we interpret your 7 posts so far (maybe 8 by now while I am typing).
What is so strange or new about one island crashing - or shall we say server.
Yes, servers DO crash but is that a noteworthy affair to be posted about? I am all for banning IP addies and if it was just for the simple reason of preventing users such as yourself from constantly spamming the blog and consequently depriving others from posting to it.
September 8th, 2007 at 12:16 AM
Oh yes and please WARN people when inventory may be lost. It would be pretty decent to do so.
Maybe we should start a jira topic about this one!
September 8th, 2007 at 12:25 AM
hmmmm….
as every time there is such a blog, most people are just here to rant and express their anger and grief to the Lindens.
We all suffer from bugs, costy losses, or humiliations like avatars unproperly rendered. But don’t you realize, all, that SL is in full development, with an unexpected growing stress on resources? That SL is all the time at the limit of what we know to do, at the verge of collapsing? Most of you here obviously don’t know anything to the difficulties of software development or running a large computer farm.
We would all like less costly lands and more LL support (in world or for running SL), but this is simply impossible. So please avoid just ranting here, give some encouragement to the LL teams, or if you can, try to contribute.
September 8th, 2007 at 12:31 AM
09:27 am. European time. Ofcourse again you did a fine job……uh,…but how about just simply logging in? I know europeans are not that important, but it would be nice, wouldnt it?
The laggggggg is tremendous the last 5 days over several sims and now,..logging in,…..when?
September 8th, 2007 at 12:32 AM
A word to LL: often I (and many others) would like to make an interesting contributions to these blogs, which are limited to 100 contributions. But too often when we open SL, the blog is already full, with nearby 80% of just rants or short useless comments. So the idea would be to be more selective on comments, or allowing a vote-like feature. I especially suggest a button “I hate the Lindens” which would be in fact dummy, with no message sent
September 8th, 2007 at 12:36 AM
Sweet Says:
when the asset sever is in TROUBLE as soon as u know you need to WARN PEOPLE right away with an IN WORLD MESSAGE WE ARE SICK OF LOSING STUFF GET THE PICTURE YET?????????? good u fixed it but bad for all of us that continue to lose so much money in lost inventory cus NO ONE WARNS til hours after the fact… i dont know if you realize how much you hurt in world business with all these inventory losses WHO WILL WANT TO PURCHASE anything in game when its 99% chance it will just poof with out any recourse what so ever. I think its high time u all worked on this cus nothing has been done to resolve this bug yet we should NOT be losing things as much as we are we need a BETTER way to back up our stuff if you cant do it for us
99%? Bullcrap
I spent %hudnreds of RL dollars and lost maybe 2 items in 9 months. And most stuff I thought dissappeared in inventory, I’ve accidently deleted or find later in another folder. But then again I keep my inventoryunder 6000 items.
September 8th, 2007 at 12:50 AM
Yeah well done folks.
Squid has been an operational thorn in my side in RL projects in the past.
Kill it and use NetCache machines.
=)
Good report.
September 8th, 2007 at 12:59 AM
“as every time there is such a blog, most people are just here to rant and express their anger and grief to the Lindens.”
I think the biggest reason people are angry is not the lindens beeing unable to erase all bugs at once, or making it perfect, but because
* they asked many times: Don’t add voice, don’t add a fancy before the grid is somewhat stable again.
The lindens don’t listen, promised the grid would not be affected, yet it did. People are afraid even more fancies will be added, and performance will become even worse.
* Linden labs makes bugs, and it isn’t preventable 100%. But why not warn people? They can make pop ups to announce the grid will be closed, or the problems are beeing investigated, so why not warn for for example inventory losses?
Its these kind of issues which make many people react off thier frustration here more than the actual bugs themselves. Its the feeling that if they don’t rant, it will only get worse and go on this way. Its the feeling that the lindens not listen. Here is a place where at leats theres a possibility the lindens will read it and who knows, maybe change something?
I guess it would be more usefull to complain in jira.secondlife.com, but unfortunately, many many people don’t even know it exists, for for those who do, it may be too complicated. I guess thats why many people rant of here. It may not be the best way, but for many, like me, it feels like we’re at least able to say something, while saying none, will improve none at all.
Just my two cents
September 8th, 2007 at 1:01 AM
@ #40/41
gon’t target coments said by other Blog users ty
and yes, i was talking griefers, the repeat kind, ty
*bows to clapping*
September 8th, 2007 at 1:22 AM
@ 54 You lost thats for sure.
September 8th, 2007 at 1:26 AM
ok, you are targeting me now, please refrain imedeatly and ask Lindens real questions, i won’t waste one more post space on this “user to user” topicing, good day, (i’ll be posting here for 9 more years + btw, to assist and help users and answer questions or post a recomendation the “real deal human being way”)
good day and good luck to you
September 8th, 2007 at 2:05 AM
Heya…. GOOOOOOOD Work bro….
well done….
yours Cris
September 8th, 2007 at 2:16 AM
@48 ‘But don’t you realize, all, that SL is in full development, with an unexpected growing stress on resources?’
unexpected growing stress on resources? this is not unexpected anymore, it lasts months already. and: if it’s a problem with the number of accounts and the number of users online (each time when online users are more than 40.000 problems dramatically increase), why then doesn’t LL just make a break..means: no new accounts or at least no new free accounts for some period of time until they can deal with this ‘growth’
I know this is economically stupid but might at least satisfy the ones who are in game already….
September 8th, 2007 at 2:17 AM
cool….
…..where my Windlight and Havok 4.5?
o_o?
September 8th, 2007 at 2:18 AM
forgot to say ‘thank you for your patience reading my post’
September 8th, 2007 at 2:28 AM
@57 we can`t hold 40,000 online without problems occuring. Oh it holds for a time but soon after things start breaking apart.
September 8th, 2007 at 2:46 AM
Thank you for telling us. I had a lot of customers complaining to me around that time that they had their money taken without getting the item they purchased. About 5 or 6 people. =^.^= Good to know your on top of things.
September 8th, 2007 at 2:59 AM
“a small percentage of assets were being lost”
how many assets are there in total? so how many were lost, in absolute figures?
i’m fed up with constant client freeze, recurrent grid problems, asset loss, failing land transactions, several weeks of support backlog (and that’s called “concierge level” - what a joke!)
i’m leaving! hopefully LL will not go bankrupt before i’ve sold out my parcels
September 8th, 2007 at 3:02 AM
Here we go again; there’s always something wrong with SL.
Our virtual world is very graphics intensive, requiring a pretty nice graphics card and a good deal of RAM. For those of us whose computers didn’t come out of the box with these devices, we go out buy the best our budgets will allow. I upgraded my computer specifically for the purpose of achieving optimal performance while in SL. It cost me around $250.
Yet, why do we bother?
Just a little rhetorical question not intended to ruffle any feathers. I know why we do it. We put so much effort into maintaining our presences in Second Life because it’s one of the most unusual and potentially rewarding virtual experiences available. Speculation on where Second Life is headed could easily read like a Sci-Fi tale.
The path SL has traversed, however, is cobbled with technical malfunctions, instability, and user discontent. Because Philip Rosedale’s creation is so special, I have maintained my belief that the road will be paved with an increasingly smooth surface in the weeks and months ahead. But then again, I am a dreamer and an eternal optimist.
I’m pleased that the latest problem has been pinpointed and resolved. But it’s just one of many that seem to pop up every week. And beyond the technical problems, there are the troublesome issues of Aristotle’s Integrity and the recent gambling ban and the way it was executed. Both of these issues make me grateful that I haven’t spent anything but time in SL.
September 8th, 2007 at 3:06 AM
now did i understand this correctly, assets are actually fetched over http? :O
I thought it was through some other means, which would have explained the inefficiency.
But as they are, so in effect, Sim gets through cached HTTP requests the asset, and then sends to the user? That is probably one of the MOST inefficient ways i know. Only reason to stay like this sounds to be for security reasons.
It would be way more efficient to have viewers contact directly the HTTP servers responsible for assets, single IP which would be round robined to bunch of Apache + mod_proxy or Squid serving servers, which in turn, passes the requests to bunch of servers running ie. LigHTTPD ot similar ultra fast for concurrent requests HTTPD.
As in question is no dynamic data, i iwouldn’t be surprised if the assets could be server by a rather small group of servers.
Perhaps 1-3 proxy / squid servers connecting to 5-10 asset serving servers. Alternative would be to save those 1-3 servers and have all the asset serving servers round robined.
Squid is a lot of overhead. Proxying is too.
I’ve run Squid as pseudo proxy for years now, and i used to run a few proxy sites. They demand POWER.
Of course, when serving static content, Nothing beats really fast scsi array
for the dynamic actions an another cluster is needed.
Of course, i do not know the inner workings of SL grid, so i might be totally lost here
One thing LL does right tho remains: Technical issues aren’t kept in dark, but they are communicated and people know what’s going on. Good for retention and damage control
September 8th, 2007 at 3:07 AM
thanks LL, you were honest and made a blog about it. and you even explained what you did.
I’m willing to bet every person who has posted “whinge LL, Inventory LOSS moan” stuff doesn’t have any clue on just how complex a system like SL is. It’s nothing short of a miracle it exists.
Put it this way, if LL didn’t care about the residents or about SL, there wouldn’t be an SL.
September 8th, 2007 at 3:09 AM
oh and did any of you think maybe part of what had broke was the ability to send in world announcements? no thought not. just step away from your computer please, now.
September 8th, 2007 at 3:37 AM
ty for fixing it:))
September 8th, 2007 at 4:06 AM
Thanks for the synopsis Lindens, and the fact that you have delved deeper into SL’s complex system and hopefully found and cured the underlying problem.
Normally I am on here moaning, and I still do about the overall performance of the grid, especially over the last 6 weeks (this last week especially - where it has forced me to find somewhere else to play happily, because it was impossible on SL), but cookies to you all for dealing with this one promptly.
I am asking for the stability of the grid, so I don’t have to pull out completely, but my customers are now getting to the stage where they are reluctant to spend money, because if they buy the chances of them getting the goods are constantly declining. Developing new products is becoming very difficult with constant database server issues. Yes I got the designs. I just can’t get them inworld.
The shiney’s are nice, and they were already in plans before all the grid issues so fair enough to bring those on line, but I trust you are now working on the bugs flat out, and I realise that the greifers are slowing that process while you have to deal with them too.
You expressed inventory backup as a priority during quarter 3. How is that going so we can get rid of all these lost inventory issues…Oh but that won’t help will it as it is the no copy items that cause the biggest problems, and they weren’t being included.
I don’t want you to cause widespread panic, but it would be nice to know beforehand when you know of grid problems, and advise of same, so I know why I can’t rez, move, teleport, or cross sims. I would get far less mad if you were open about it (like the source code!!!).
September 8th, 2007 at 5:00 AM
Good work LL…
Thanks.
September 8th, 2007 at 5:01 AM
PLEASE INFORM THE COMMUNITY WHEN WE COULD POSSIBLY LOSE OUR INVENTORY A LOT SOONER THANKS
September 8th, 2007 at 5:19 AM
I hope this post is not too off-topic, but I need some understanding of my lag issues. Firstly I have researched the information pertaining to lag and statistics bar in the knowledge base, the former has some excellant advice, the later, well the information is sadly out of date!
I cannot understand the following, every evening when I log in (8 gmt onwards, noon pdt) and the number of users online is over 40k I am experiancing personal fps of:
Azzurra Valley ~ 7-8
Eastfield, Quietly Red ~ 2-3 !!!!!!!
In the morning like now (4am pdt, 30k users online)
Azzurra ~ 25-40
Eastfield etc ~ 12-14
So my questions are why does 10k extra people online make such a difference in my fps? I cannot see anything in the other stats that help me explain my low fps. My packet loss is minimal, my bandwidth is similar, as is time dilation & time figures. These are all class 5 servers (in the case of Eastfield on the nautilus continent, these new sims are not built up). In the evenings I cannot move without jerking and freezing, typo lag is terrible and scripts take seconds to respond.
Recently visiting Izi mediaworld, Islander Resort, i find I am consistant getting readings of 40-50 fps, day or night!
So why my lag? Why so much difference in fps between similar sims?
On the issue of inventry loss, yes I have lost items and they always reapear when pulling one item out of lost and found,often several “missing” items will re-appear. People new to SL need to be aware of this. Sometimes its not just items in lost and found but also in other folders that “hide” your missing invent. not saying this is the case for everyone, but if like most here I know our inventries are getting out of hand, then an hour or so occassionally tidying up and pulling objects out will work wonders!
Griefers! No we should not have to put up with them in sl, I do not put up with them in rl so why should I here? What is the point in reporting abuse when two weeks later LL has done nothing more than send me a confirmation email that it is investigating thoroughly and and will let me know when its resolved. In the meantime the abuser with 5 bans already can continue abuse. Oh and the police blotter is a joke, thanks LL for protecting the identity of the griefers, yes I guess they have a right to anonymity, and a few days off on their alt avis will be a nice chance for them to plot their revenge!
smiling still, its a brand new sl day and the suns shining out there! another day of unexpected happenings, (if I can log in!!)
September 8th, 2007 at 6:42 AM
“This bug will mot recur” how rarely i read those words on this blog.
usually Bugs are not fixed but “resolved” I hope to read this trend continues
This time i only seem to have lost one building.
September 8th, 2007 at 7:06 AM
Ok great, so some grid monkey messed up the permissions on something that was previously working, and as a result, sigificant asset losses occured, among other issues.
As this outage was your fault, are you now taking responsibility for those asset losses, which cost your subscribers time and actual money? Last time I reported such an asset loss of over L$8000 in purchases, it took 22 days for a reply to get back to me, and that reply was completely unsatisfactory: “Clear your cache; case closed.”
I realize that inventory losses are a complex and ongoing issue, yet the unwillingness to take any kind of responsibility for them is a significant concern. When things like this happen, I can visually see traffic in malls that I own dropping to almost nothing.
Also, what the heck happened to the name search function? Searching for a name now brings up an A-Z list on each page and subsequent pages, rather than a descending list across pages. We have to search each individual page now and hope they are in there.
September 8th, 2007 at 7:09 AM
Also the teleporting is become problemsum again. Not only Lag issues but we are now become less stable as the day goes on.
September 8th, 2007 at 7:19 AM
Thank you guys. Very well done.
I liked this post! You reported us the bug cause, what you did to repair and seems it is REALLY resolved.
I suggested before to let us move the tolls windows and IM windows out of the game screem… allowing us to use one monitor just for the game without those sub-windows.
Today I want to suggest you the release of an SLight Viewer. An viewer with only the very most BASIC FUNCTIONS. This is something simple but very welcome.
September 8th, 2007 at 7:20 AM
This has absolutely nothing to do with a response to whoever’s post may be above mine but….
I love reading these blogs it makes me feel so good about the people I deal with in SL. It reminds me how lucky I am to have good friends that know how to work together as a team, stay on topic, and participate as adults toward common goals.
I thought the initial blog post, and the response to the problem they had to deal with was first class!
What goes on after the posting in here is hilarious and very fun to read. Some genuine people that I would be honored to add to my friends here but a lot that you have to wonder if they are even aware of what the topic is or what they are talking about. And then there are those that I think live just to complain in response to these blogs an unmistakable sign of a very meaningless life but to me funny all the same.
Thank you so much LL for this very informative blog and timely response to a potentially very harmful problem!
September 8th, 2007 at 7:31 AM
Thank you for allowing comments….this makes the standard “Thank you for your patience” comment MUCH more tolerable
I know this is all an experiment hence “Linden Lab” and not “Linden Company” but I agree which a previous comment….how is it that things like permission on a very important script just change?
September 8th, 2007 at 7:53 AM
Слушайте пидарасы, вы когда нибудь чего нибудь пофиксите нормально сколько блядь можно? Сколько из-за вас мудаков потеряно, сколько вы своровали, никаких извинений, никакой компенсации, никакой техподдержки на платном аккаунте, ен говоря уже про бесплатный, часа невозможно работать активно в SL чтобы не случилось какого нибудь бага, причем давно известного, и отосланного вам 20 раз. Вот в очередной раз за 15 минут до важной встречи опять ваше дерьмовая сеть ни хрена не работает. ДА КОГДА ЖЕ НАКОНЕЦ ВЫ ИДИОТЫ ТАМ ВСЕ ПОДОХНЕТЕ!! чтобы упокоились ваши жадные кривые руки в мире, и на работу взяли нормальных программистов.
September 8th, 2007 at 8:16 AM
Thanks guys and gals for that detailed explanation. I know that was some hard, hair pulling work.
to 19 and the others: this is a grid networked computer system, Things happen. Things break randomly when heavily used. Image driving your car a 1000 miles a day. That 10, 000 mile tune up is at the end of the week suddenly and 3 months later you are a 100K miles. Car has to be overhauled.
The core of SL, servers and stuff has to fixed on a habitual basis. Whining about your pixels being inaccessible is disingenuous. It will happen. It sucks, yes. but it will happen.
As to the sony comment. I hope you are on the same IP block as the griefer once (IT can and has happened) so you can experience the stupidity of that statement. Hardware blocks are the only effective way to end griefing. And that takes time.
Oh and if you are not a premium member, you have no support because you aren’t paying for it. This is’nt a support page. No one is reading it to fix your problem. Don’t like it? There is still open.