UPDATE: Failed Teleports and Other Slowness
Sunday, November 19th, 2006 at 1:27 PM by: Robin Linden[PST 2:00 PM] After some work on the databases it appears that everything is back to normal. We’ll continue to keep an eye on things. Thanks for your patience.
[PST 1:24 PM] We are investigating reports of failed teleports, faulty L$ balances, and clothing not appearing. The problem seems to be tied to heavy load on the database. The ops team is on it, and we should have an update for you shortly.
November 19th, 2006 at 1:31 PM
And my object inventory lost a script I’d spent hours writing.
Thank you for your excellent quality assurement, Linden Labs.
November 19th, 2006 at 1:33 PM
Hiya.. question, I thought I heard Philip say in the town meeting that the servers were now 1/2 way across the country as opposed to being local to LL. Could this be a “redline” syndrome reason? If the provider changed, when was it and could that affect the load/bandwidth problems we’re experiencing?
November 19th, 2006 at 1:33 PM
While I and many others have a vested interest in SL’s continued growth, perhaps it is high time to limit the number of people who can log in to the grid at once, until the scaling issues have been solved.
November 19th, 2006 at 1:37 PM
Too many anticopybot spamming devices? Hmmm.
November 19th, 2006 at 1:38 PM
As much as I hate to say it. I agree with April. if I log in during the late morning 10-11am -0600GMT with 6000-7000 loged on there are no issues (NONE! NADA)
anything over 10000-11000 it starts. and 14000+ forget about it.
November 19th, 2006 at 1:38 PM
You mean failed teleports, faulty L$ balances, clothes not appearing, objects not rezzing, attachments not attaching, not being able to create any inventory objects at all… this is all still related to asset server problems. It is upseting that it has taken more that three hours of dealing with this for there to be even a minor update to the blog.
Nothing has changed since these problems started occuring weeks ago and they are only getting worse. During the week it’s not as bad, but you’ll notice they keep taking a nosedive on weekends when more people are logged in. Something big is going to need to change before this will even start to get better and making patches here or there is not going to help.
This is a major issue and concern for the residents of second life. It is right now, and has many times in the past, made SL completely unuseable. It is no more than a glorified insitant messenger at the moment.
November 19th, 2006 at 1:39 PM
I have also noticed issues with editing objects - they will seemingly move around a bit at random, rotate themselves back to their original position, and not correctly display their contents. I am assuming these are also caused by high database load?
November 19th, 2006 at 1:41 PM
I got the ‘downloading clothing’ message before teh announcement. After 10 minutes and still not able to see myself other than my attachments and I decided to relog. Saw the announcement. Tried to log in and get a login failed message: unable to find a simulator. So now I can’t even get on and with a message I’d never heard of before.
November 19th, 2006 at 1:42 PM
**failed teleports, faulty L$ balances, and clothing not appearing..**
I had this happen to me in RL a few times, and my doctor prescribed Thioridazine.. I’m fine now except for the occasional episode of street corner ranting.
November 19th, 2006 at 1:43 PM
Ugh, downtime bleh! but hey everyone who’s gonna get angry just look at the bright side in an hour + we’ll have a much better SL where things load and less of that ugly missing texture. Also less lack of teleporting lately thats been a huge problem
-GOOD JOB SL!
November 19th, 2006 at 1:44 PM
Park went down as I was logging in. They told me that downed sims are not a priority right now due to these database issues.
GRIEFERS OF THE WORLD UNITE! TAKE THE SIMS DOWN NOW!
After all, if they’re fixing database issues while you’re shutting down the grid, they won’t do anything to bring them back up until after the database is repaired.
Dee Linden told me so herself!
Lynn Kukulcan
November 19th, 2006 at 1:45 PM
Inventory is also not showing what is in the folders. As for limiting the number of people on the grid… that isnt going to solve anything. All it will do is make people leave SL completely. I know if I could never get on when I wanted to… I would sell or abandon my land, drop my premium account, and throw in the towel on SL as any kind of entertainment or business venture. I am sure there are plenty of others who would do the same.
LL needs to fix the issues, and fix them fast. I think we have all been patient long enough. The constant problems are getting old.
November 19th, 2006 at 1:45 PM
I had nearly 200 items fail to make it to my lost and found folder after a property sold. Very bad. This happened yesterday (Saturday). Wonder if it is related?
November 19th, 2006 at 1:46 PM
Distilled 1 - I’ve also noticed that once there are over 13-14K users on, SL is very probmatical. Scaling??
April - I’m not at all against free accounts, but will not be happy if free accounts can get in, and paying accounts can’t.
November 19th, 2006 at 1:46 PM
I assume the lack of warnings in Sl abt closed logins, and that about maintenance is also a result of database load?
November 19th, 2006 at 1:47 PM
Lol Davis! Yeah when i was working on stuff i too found it hard cause they would zip back to a previous position. Was kinda funny but after a bit it got old and frustrating. Also some of my stuff that was nocopy that I tried to rez didn’t rez, got a failed rez thingy and then it wasn’t back in my inv. Hopefuly it’ll be there after this update or I can contact the maker for another. SL been very slow and with that rolling update they did it didn’t seem to help none. I dunno what’s going on but I remember a better, smoother running SL priror to Release 12. Hope you all can get it sorted out soon. And these random sim crashes have to stop also sometimes it’s like 3 or 4 in a row.
November 19th, 2006 at 1:48 PM
We hit 15,000 today. A new milestone, but definately stressing the system.
November 19th, 2006 at 1:48 PM
It is upseting that it has taken more that three hours of dealing with this for there to be even a minor update to the blog.
Nothing has changed since these problems started occuring weeks ago and they are only getting worse. During the week it’s not as bad, but you’ll notice they keep taking a nosedive on weekends when more people are logged in. Something big is going to need to change before this will even start to get better and making patches here or there is not going to help.
This is a major issue and concern for the residents of second life. It is right now, and has many times in the past, made SL completely unuseable. It is no more than a glorified insitant messenger at the moment.
November 19th, 2006 at 1:48 PM
I have a question for Linden Labs… seems alot of the performance issues lately have been linked to overloaded/overworked/fubared databases. Got a simple question… why didn’t anyone think about upgrading before all the recent press hype, and the change in account creation policy? They his over a million accounts, and now nobodies SL experience is worth it, freebie accounr or paying customers…. Does this make good business sense to anyone?
Signed,
Very irritated and dissatisfied paying customer
November 19th, 2006 at 1:49 PM
Also Does having like 20k items in your inventory not helping at all?
November 19th, 2006 at 1:50 PM
I have simular experience as Distilled1 with -6GMT morning lag levels. They just don’t really exist. However, the past couple of days there seems to have been a steady decline even in the usability during that time.
An added dose of litium may help controll the episodes of street corner ranting. Did for me anyway. =P
November 19th, 2006 at 1:50 PM
April is right. You need to start taking care of your current costumers, instead of ignoring us letting these problems happen by chating ‘more users more users more users’ when your not ready to give good service to the ones you have.
I caneled my paid account because of stuff like that.
November 19th, 2006 at 1:50 PM
@ralph:
SL will undergo the “ICQ Effect” anyways: IE, you give people something free until a large demographic depends on it for day to day activities, then you start charging them for it or forcing ads on them with the concept of “well if you don’t like the ads you don’t have to use it.”
November 19th, 2006 at 1:50 PM
Yeah, I’ve been experiencing some MAJOR issues similar to those. >_>; The Search option doesn’t work, teleporting is nearly impossible, and in some cases I was actually unable to even move, and my avatar was in an unusual position as if typing and trying to fly at the same time. ._.;
However, I notice that the Teleporting issue and being stuck SEEMS to take place in more populated sims. Things like Searching and Teleporting into populated sims are what appears to be the cause of the database problem, correct me if I’m wrong. o.o
Perhaps an evenly distributed Sim user-cap would be a possibility? >.> I’m not VERY familiar with how the grid works, but it IS a thought.
November 19th, 2006 at 1:51 PM
*Blinks eyes*
Is it two weeks ago?!
Dang… thought I could skip through time. Oh well, back to everything other than SL.
November 19th, 2006 at 1:51 PM
I personally am tired of the lagging and loading and L$ balance problems lately, its getting to be so annoying that i just want to log off for day. And i also agree with April because i get online during the night, and there is hardly anyone online. It is alot easier to move around and load and my favorite it’s lag free.
[Please Fix It].
November 19th, 2006 at 1:51 PM
Park’s been down a half an hour. That’s no database issue. One time, it was down at least two hours in spite of being reported repeatedly, and no one brought it back up. No database issues at that time.
I guess crashed sims have no priority at all for repair. I kinda’ hope it’s still down this time next month, so I can scream that most of my land holdings have been unavailable to me for a month so I shouldn’t have to pay land tier for them.
Lynn Kukulcan
November 19th, 2006 at 1:52 PM
@ralph doctrow > Distilled 1 - I’ve also noticed that once there are over 13-14K users on, SL is very probmatical. Scaling??
Agree…we get a database load issue with only 1% of the total number of residents logged in? Nearly 1.5 million online accounts and only a few over 15,000 logged on. Seems like SL is not an extensible model to me.
Skip
November 19th, 2006 at 1:52 PM
Quote - Rose Bradley
“I caneled my paid account because of stuff like that.”
Im not too far behind you Rose.
November 19th, 2006 at 1:53 PM
oh joy, another weekend problem. I pretty much can count on the fact if it is a weekend, there will be a problem and the grid will be offline at some point. Note to self, do not plan anything in game on the weekends. LOL
November 19th, 2006 at 1:53 PM
Gosh, who could have forseen these kinds of stress problems on the servers? Certainly it isn’t a result of Linden Lab making registrations free, promoting the thing to death, and failing to plan for this?
I just hope hope hope hope HOPE it gets fixed.
November 19th, 2006 at 1:53 PM
I got stuck in a Welcome Area for 20 minutes! Finally I decided to fly 6000miles to my home. It was a verrry long journey.
November 19th, 2006 at 1:54 PM
Don’t we all wish there was an SL equivilent of Geek Squad? I took in my crashy comp yesterday, now it works like a charm.
November 19th, 2006 at 1:55 PM
i have had not been able to see my clothes or others for about 2 weeks and no one has answered me until now and when did, what they suggested it didn’t work. my computer has been reformatted in the middle of this problem and is not my issue but the server on sl. it is getting annoying that i can not do anything. is it going to be fixed?
November 19th, 2006 at 1:56 PM
And how long have they been “working” on the high packet loss issue??? Seems to me there are a lot of problems being “worked on” but not many getting fixed.
November 19th, 2006 at 1:56 PM
Number of items in invenrtoy doesn’t seem to affect things unless you clear cache. What does help is to remove ALL attachments, HUDS, and anything but a pair of pants and shirt. Then fly up a hundred meters, change draw distance to 64 and try to TP. Seems the amout of clutter around you and on you makes a huge difference.
November 19th, 2006 at 1:58 PM
Um.. i cant see myself at all .. on my thing… can someone hellpp?
November 19th, 2006 at 1:59 PM
I keep hearing database in it’s singular form. I would have thought there would be multiple databases each on a high speed server. One for objects, one for textures, one for scripts and so on. Keeping everything in one database is just asking for overloading.
I would also think once one servers database is to the point of overloading you could stop it from receiving more new things and start up another server to carry the load. Seems to me the only way you could scale. Not sure how to code but each object could have a database ID so it knows what server to look for so it doesn’t have to go through all the data base servers.
I have no clue how SL is setup though. However it is it doesn’t work after 13,000 connections. I started to notice considerable slow downs after 8,000 connections even in sims that have no other people in them. After 10,000 connections it becomes hard to play. Usually after 12-13,000 I stop playing because of lag and slow load times.
I am assuming these problems are why other 3D chat and MMORGs do not allow so much content creation and limit the textures and scripts and AVI detail. Maybe the only way they can grantee smooth service.
November 19th, 2006 at 1:59 PM
That’s all she wrote guys. Grid’s closed. For all the “looking into it” there has been no solution yet. Here’s to hopes the new code for troubleshooting helps.
November 19th, 2006 at 2:00 PM
Nothing has changed since these problems started occuring weeks ago and they are only getting worse. During the week it’s not as bad, but you’ll notice they keep taking a nosedive on weekends when more people are logged in. Something big is going to need to change before this will even start to get better and making patches here or there is not going to help.
This is a major issue and concern for the residents of second life. It is right now, and has many times in the past, made SL completely unuseable. It is no more than a glorified insitant messenger at the moment.
November 19th, 2006 at 2:00 PM
@ Atlwolf, That is great advice altwolf ty. Also can I ask please. So cache should be cleared out ? every what ? daily.
November 19th, 2006 at 2:00 PM
ok i’ve seen all that, but how about this. i found several objects floating around my house…that i used and deleted over a week ago. lol its getting strange, or is it me?
November 19th, 2006 at 2:01 PM
Atlwolf Says:
November 19th, 2006 at 1:56 pm
Number of items in invenrtoy doesn’t seem to affect things unless you clear cache. What does help is to remove ALL attachments, HUDS, and anything but a pair of pants and shirt. Then fly up a hundred meters, change draw distance to 64 and try to TP. Seems the amout of clutter around you and on you makes a huge difference.
—
I did all that and couldn’t TP in to Park. I got to Junghee, right next to Park, and Park was a GREAT BIG HOLE!!!
None of this is any good if your sim is crashed. Also, the fact that the Lindens aren’t paying attention to the grid atm means that it’s a great time for griefers to run rampant and take it down, so their precious baby would fall apart right in front of their eyes.
What’s the point of having an inventory database when the sims are down?
Lynn Kukulcan
November 19th, 2006 at 2:01 PM
Not just you hope.
November 19th, 2006 at 2:02 PM
Obviously, this shows that something needs to be done on LL’s end of things. Upgrading servers, or limiting access to a certain number of people until these issues are fixed. I also don’t have any issues depending on what time of day I log in. When there’s hardly anyone on, there’s NO issues.
November 19th, 2006 at 2:03 PM
If overload of users is making problems, I agree on to limit access to the grid until the problems are solved. And I am also agree on that one should consider priority access to paying members. As a private user, I spend thousends of dollars each month, and to be honest, there are far too many problems in SL to be accepted. I know there are huge problems to be solved by Lindens, and all salute for that, but as a client, spending awfull lot of money in the game, one feel really stupid. If Linden not can handle the growth, they should consentrate to deal with the existing members. Size not allways matters.
November 19th, 2006 at 2:03 PM
“instead of ignoring us letting these problems happen by chating ‘more users more users more users’ when your not ready to give good service to the ones you have.”
No kidding. That was the philosophy of SOE with Star Wars Galaxies… and anyone who has been following online gaming at all has probably heard of that fiasco….
November 19th, 2006 at 2:05 PM
i’d also like to note the inabiltiy to login. sometimes when my game crashes, and i relog, a message states “the system is logging you out right now, please wait till #:## time” which would work afterwards. but lately when i try to relog at the said time, the message coems up again and again, with increasing increments of time. it did this for a couple hours before i could finally log in again. today i couldnt at all, except breifly in the morning, before the game crashed again.. >_>;
November 19th, 2006 at 2:06 PM
Add to the list of anomalies - objects from neighboring parcels scattered around to other parcels. I’ve got Casino equipment in my home…
November 19th, 2006 at 2:07 PM
finally a blog post about this… I’ve been having isues ever since late friday :/
btw, woudln’t killing sims actualy help right now with easing the load?
November 19th, 2006 at 2:07 PM
Jaydee, I work on databases so I assume SL is configured as a cluster. I believe there are 5-10 different “regions”, and each one has it’s own database. Even a cluster has limitations since all transactions eventually have to be synched in the other databases.The problem is not 15,000 residents specifically, it the sheer amount of information that has to happen everytime someone TP’s. Not sure what the answer is. They need a good systems architect too look at thier overall design so it can scale.
November 19th, 2006 at 2:08 PM
I remember a time when we used to get messages in SL telling us of problems and when they were sorted.
I that not being done anymore ?
November 19th, 2006 at 2:08 PM
Yes, I administrate an SL town that even during peak time used to have no lag whatso ever. Lately even my place as system resource friendly as it is shares many problems…..
when editting items:
Reverting Textures
Revertiing Angle and Position on objects
The oops where in the !! did it go syndrome
Also Lighting isn’t working correctly… O.o
All around…. Oh noes……
Sniffle sob* and oh crap!
Anyways numbers of problems that I didn’t experience when I first started 2 months ago when I upgraded to premuim on Second Life,
Get is fixed soon ^^,
Perefim
November 19th, 2006 at 2:09 PM
At least, temporarily (!!!) halt the free new user signups! Charge 10 USD, or 5 USD/month, whatever, but try to throttle the load increase, if you can’t handle it, Lindens!
And please, if you want to cut the L$ ’sources’, first cut Supply Linden L$ selling, only then the instructor wages!
November 19th, 2006 at 2:10 PM
Yes, why not limit access to premium members until this is sorted?