A big change you’ll barely notice
Thursday, December 21st, 2006 at 10:47 AM by: Zero LindenThe next release will contain a major piece of new functionality that, if we’ve done it right, you won’t even notice. I can hear you all exclaiming “What?!?!”, and warming up those comment forms… But hear me out.
We’ve just completed and passed the first phase of the “sim-caps” project. As Cory alluded to in his Town Hall, this project enables the simulator and viewer to communicate using a collection of standard technologies: TCP, HTTP, SSL/TLS and XML. These offer a better match of performance characteristics for much of the viewer communications, and benefit from the many years of development these technologies have had.
So why won’t you notice? Because for this release we took an extreme view of writing software in a way that has very low risk to the grid:
- Only a small number of operations will use the new system at first
- The existing messaging code for those operations has been left in place
- The grid can be switched, on the fly, per operation, between using the old messaging system and the new.
This last technique is notable: It represents an engineering approach to designing with the goal of not taking the grid down. We will be building more new functionality with this kind of “live fallback”. And how do we know the fallback works? The “sim-caps” code has been deployed for about a month - we’ve just been using the live configuration to keep it turned off!
While this change is small, it is the first step in the very large set of changes Cory talked about yesterday. The next steps will involve greater and greater functionality and you’ll begin to see the benefits in how things function. The end of this path is a day when we can update the grid, viewers and simulators, in pieces, without ever taking it down.
If you’ve read this far, you must have a real tech. fetish, so let me indulge you a bit more!
The “caps” in “sim-caps” stands for capabilities. This is a well known design pattern whose biggest success has been web cookies. Now before you all jump on that, realize that cookies, properly implemented, are what make the internet go ’round. It is how every major site with accounts keeps you logged in. We’ve taken much care in our capability design and use strong security (SSL/TLS). And unlike cookies which are “all-or-nothing”, our capabilities are for small pieces of functionality, valid for short periods of time. If one is compromised, it won’t compromise the whole system or even a large part of it or even your whole account. (See these entries in Wikipedia: Capabilities — HTTP Cookie)
The messages themselves are built upon LLSD, our own developed technology for encoding structured data. This is, in turn, serialized to XML and sent over the wire. While this does increase the number of bytes it takes to encode messages, it offers a degree of forward and backward compatibility we don’t currently have with the current message system. (Which is why you all have to download new viewers every time we update.) Messages where this overhead is significant will still use the existing system. Fortunately, those messages are well established and haven’t changed in ages.
The first set of operations that will use the new messaging system will be: teleport, region crossing, some map data, and asset uploads. This will almost certainly change the timing of teleport and region crossing - in some cases slower, in others faster.
If you like peeking under the hood (don’t mind the grease), then look at the Second Life log file. Today, you can find this line:
2006-12-21T04:39:08Z INFO: BaseCapabilitiesComplete::error 499: STATUS_EXPIRED
Which shows that the viewer is trying to get the capabilities, but that right now, the server isn’t giving out any. Hence, all functions are running over the old message system.
When you open the map, the log will show:
2006-12-21T05:40:36Z INFO: LLWorldMap::sendMapLayerRequest via message system
2006-12-21T05:40:36Z INFO: LLWorldMap::processMapLayerReply from message system
2006-12-21T05:40:37Z INFO: LLWorldMap::processMapLayerReply from message system
This shows map layer data (which is a small portion of the map data) being requested via the existing message system. Notice that it takes multiple replies to get the data back, as the data doesn’t all fit in a single UDP packet. In the future you’ll see “via capabilities” in the request, and only one reply, which will say “from capabilities”. This is because over HTTP we can deliver the whole response in one go. That in itself is a big win for many of our messages.
Hope you’ve enjoyed this rather technical romp. I’m spearheading the whole messaging infrastructure project and will try to keep you informed as we hit major milestones like this…. even if you’ll barely notice!
December 21st, 2006 at 10:58 AM
soudns interesting I for one hope it works well as im sure the techs have put there soul into it
December 21st, 2006 at 11:09 AM
This is *BIG* *BOLD* *ENCOURAGING* *NEWS*! Please keep us updated. We really really need to hear these things.
December 21st, 2006 at 11:12 AM
I’d like a drop of that LLSD on a sugarcube please.
December 21st, 2006 at 11:13 AM
So, can we then create sim content that can drive internal objects from external data? Possibilities include simagng, data tracking, etc.
December 21st, 2006 at 11:13 AM
Awsome news!! Please keep us updated on this!! Please….pretty please???
December 21st, 2006 at 11:16 AM
is is normal that i dont even understand it?? *blonde moment*
December 21st, 2006 at 11:17 AM
Thank you. I would like to see more focus on back-end development. The UI is fine, the “Focus” client I’ve been thinking of as the “lack-of-Focus” client. People want improved performance, please continue in this regard.
December 21st, 2006 at 11:17 AM
Zero, you know that most of the users will read your post and just go “huh?”, don’t you?
Some of us will nod sagely, though, understanding what a wonderful step this is. Those who don’t know what you’re talking about *will* notice changes, though. Functionality can now creep into SL, almost undetected. We won’t experience grid-wide outages as upgrades are installed. We won’t have to deal with the chaos of multiple, inteacting changes mucking up things. This is truly a positive direction, and LL is to be aplauded for doing it.
December 21st, 2006 at 11:19 AM
Awesome! Well done so far… looking forward to the continuations!
December 21st, 2006 at 11:23 AM
Right on. I mostly understand you… please keep posting this info when you can, and we will try to learn. Thank you!
December 21st, 2006 at 11:25 AM
Sounds good. So when will you have the hood down on the Beta Grid so we can take her for a few checkout laps before she has to tour Daytona.
December 21st, 2006 at 11:26 AM
Sounds great! Glad to see more standards being put in place. I know you say we won’t notice, and you’re probably right, but after more of the messaging is cleaned up and what used to require multiple communications (like the map example) starts requiring only two (request & answer), we’ll surely (hopefully?) start to see some improvements in communications (less lag).
Thanks for keeping us updated on things like this. You could have just made the changes and not said anything about it, but I for one appreciate it and I do enjoy learning more about what makes SL tick.
Thanks!!
Iron
December 21st, 2006 at 11:30 AM
Yey Caps! I look forward to seeing this implemented, but libsl will have some catching up to do.
December 21st, 2006 at 11:34 AM
As a software architect myself, I totally get what you’re doing, and I for one am glad to see it. This is the kind of thing that will provide the reliability and performance that make for a good user experience, and enable other features to be developed on a solid foundation. Keep up the good work!
December 21st, 2006 at 11:36 AM
Very cool, from the looks of things this is a big step in the right direction. Kudos to the developers. I hope the transition is relatively smooth. Switching region crossing and teleport away from UDP sounds like a good idea considering the sometimes unreliability of that protocol, that has me thinking that this might help with the failed sim crossings we sometimes see since the other protocols you mentioned are more reliable, correct me if I’m wrong though.
December 21st, 2006 at 11:44 AM
Glad to see the first deployment of this system. I remember reading about it after the SLCC and being very encouraged by the idea.
December 21st, 2006 at 12:06 PM
This is fantastic news, and I applaud you linden for it. This is a very positive step indeed. Bravo.
December 21st, 2006 at 12:12 PM
If textures can/will be transferred using HTTP, does that mean proxy servers could be set up to make a cache of them and reduce the loading time or the load on the SL servers?
December 21st, 2006 at 12:12 PM
Next up, primcap for scrolling text on a prim, which will of course make reference to the venerable termcap database
(TERM=tortured-torus)
Good luck with this update - just dont roll it out before xmas!
December 21st, 2006 at 12:16 PM
This is definitely encouraging news, and I absolutely LOVE reading tech posts like this. And as technical as these posts tend to be, LL does a consistantly good job at translating this stuff into plain english without dumbing it down too much. Now, if only the legal, management, and financial people had that talent….
Seriously, I’d love to see more posts like this. I think I speak for everyone when I say that we all love getting glimpses of just makes SL tick. Keep up the good work. =)
Take care!
- SDW ^..^
December 21st, 2006 at 12:22 PM
For you non-techies, “UDP” is the “Unreliable Datagram Protocol”. It’s a way to moving data across the Internet that is, as the name suggests, unreliable. The advantage is that it’s low overhead, but trying to *reliably* use it requires adding some additional processing on top to handle situations like dropped or out-of-order messages. Once you do that, you’ve chewed up the advantage UDP offers, so you might as well have used a reliable protocol to begin with (exception: sometimes your protocol doesn’t mind dropped or out-of-order packets, in which case, UDP is the best solution, but that’s not really the case here).
Thus, the move away from UDP to established reliable protocols is a good thing.
December 21st, 2006 at 12:39 PM
Very, very very encouraging news. I was dumbstruck when I originally heard that LL was using UDP for everything, given its unreliable nature (UDP guarantees timely delivery attempt, but not reliable delivery, TCP the opposite) and much of the messaging system seemed to be made up of attempts to add the delivery reliability of TCP to UDP - an approach we’ve all seen the down-side of.
Thanks for moving to tried and true methods for the things that need them most. Keep it up LL!
December 21st, 2006 at 12:54 PM
Score!
Good job, LL. Now SL can be webified.
Can you make it run over port 80, too?
How about XML for scripts?
This is really awesome!
December 21st, 2006 at 12:56 PM
YES! YES! YES!!!!!!!!!!!!!!!! I LIKE IT.
December 21st, 2006 at 1:06 PM
Very good news ! I remain cautiously optimistic.
December 21st, 2006 at 1:07 PM
So to sum it up , on day we wont have to sign off when you at LL are updating the grid or fixing things?, Does this mean we wont have to down load new updates every other week? or do will we still be doing that but just having to upload the next time we sign on , assuming we sign off, some of us are online a tad too much ….we know who we are, wink…; )
December 21st, 2006 at 1:09 PM
I’d like to see this tech-stuff blogged more often.
THX
Zero L.:
I’m glad to see that Linden is capable of making such imo drastic design changes.
If you continue to improve at that rate you might even make it to the “Snowcrash” level of quality.
Kat S.:
You don’t have to know how to build a car to drive it.
Gaius G.:
Good try at explaining UDP. The wikipedia entry isn’t half bad either.
SL isn’t a F[irst]P[ersion]S[hooter] where every single milisecond of data roundtrip counts that bad. For a game like Quake UDP makes sense, but i was suprised that SL use(d) UDP at first.
(Could have mentioned that the “reliable” thing is called TCP
December 21st, 2006 at 1:14 PM
This is awesome information. I agree you should be putting things like this out earlier. It would have made the problems we were enduring more understandable. It seems a lot WAS being done under the hood to make things better. You guys just weren’t tooting your horn about it.
Keep it up LL.
December 21st, 2006 at 1:16 PM
*I can hear you all exclaiming “What?!?!”, and warming up those comment forms… But hear me out.*
Yes, listen you bunch of whiney buggers! We don’t have a bunker mentality!
Anyway, this seems to be positive change, I’ll wait to comment on it until I get to use it, I just couldn’t resist a jab when that’s how the blog starts out.
December 21st, 2006 at 1:17 PM
UDP actually stands for “user” datagram protocol (not “unreliable”) — though it *is* unreliable.
There are of course many valid reasons for using UDP over TCP, because not all communication need to be reliable. Frequent position updates, for example, could benefit from UDP. Live audio and video streams are typically UDP based.
When networks are congested, using TCP could actually make the user experience worse, not better. That’s because TCP tries hard to deliver packets, even when those packets are no longer relevant as time passes.
Often it’s better to just drop packets rather than insisting on having them reliably sent. In fact our entire telecommunications infrastructure is based on this idea (e.g., ATM networks).
-peekay
December 21st, 2006 at 1:35 PM
[quote]
If you’ve read this far, you must have a real tech. fetish, so let me indulge you a bit more!
[/quote]
nope, I was hoping I’d start to understand what the hell you were talking about.
(Needless to say: I have NO clue)
December 21st, 2006 at 1:45 PM
Perhaps I overlooked it, but when are you going to start switching in sim-caps?
December 21st, 2006 at 1:49 PM
Posted at 10:47 PST. Why weren’t you all at the snowball fight? It’s way good fun. Especially you Zero. Hmmmmm?????
December 21st, 2006 at 1:51 PM
I have a “real tech fetish” but still was unable to masturbate to that.
Please rephrase it to be a bit more erotic, and maybe add some danger and intrigue. A little romance and foreplay would be nice too.
And would it kill you to take me out to dinner once in a while? I’m sick of hanging around the house day after day while you go out getting techy all over town.
December 21st, 2006 at 1:54 PM
What worries me is the “if we’ve done it right” part. We all know how efficient LL is about testing before a release…
December 21st, 2006 at 1:58 PM
I am happy to hear this news, but I have a feeling we WILL notice this “barely notice” change. No offense but your track record for “patching” has not been very good and I don’t have faith that this will go off without a hitch.
December 21st, 2006 at 1:59 PM
This is the kind of technical detail that I just eat up whenever I can find it. Nice!
The technique of using XML and TCP kind of sounds like the Jabber protocol…might this have some implications for the “Jabberization” of IMs that’s been promised in the past?
December 21st, 2006 at 2:02 PM
database is down again. again. AGAIN. AGAIN. what good is this news when qwe cant do anything because the database is always down and/or our expensive stuff has vanished? Lindens need to be held financially accountable for lost inventory since it is their ineptness that is causing all of the issues.
December 21st, 2006 at 2:03 PM
Quote
—
This will almost certainly change the timing of teleport and region crossing - in some cases slower, in others faster.
—
Me no likey the sound of that >
December 21st, 2006 at 2:06 PM
it will no doubt make sl even worse and more unreliable than it already is now. its become useless. and everytime they come up with a new brilliant idea it makes sl worse than ever.
December 21st, 2006 at 2:06 PM
re: Erbo - yeah, I’d love the IMing to be “jabberised”… that way I can keep in touch with people inworld at places like work, without using the full client (which is a no-no)
December 21st, 2006 at 2:08 PM
This has the right sound, keep up both the development and the communication please.
December 21st, 2006 at 2:09 PM
finally an update to the grid almost as big as flexi…but without as much conversation among us all..then again there wernt this many residents back then…
December 21st, 2006 at 2:17 PM
Angela Glitter Says:
December 21st, 2006 at 2:06 PM PST
it will no doubt make sl even worse and more unreliable than it already is now. its become useless. and everytime they come up with a new brilliant idea it makes sl worse than ever.
Sorry, sunshine, but you’ll notice most of the comments in this thread receive this news positively, probably because most of us understand it. Don’t sweat it though, I’m sure you’ll have the chance for a negative energy pile on sometime soon…but this isn’t it.
December 21st, 2006 at 2:20 PM
YEAH!! Just what we needed!
And thanks for taking the careful approach on deploying this.
P.S.
Please do more of these “Under the hood” posts!
December 21st, 2006 at 2:28 PM
The “if we’ve done it right” portion hasn’t been working out too well in LL’s favor lately. Not everything that promises a big move forward has had that effect. I’m a little skeptical and I’m sure I’m not the only one.
When you mention the map layer data not fitting in a single UDP packet and moving to HTTP to deliver the response all in one go, you are mixing two different levels of network communication. UDP (and TCP) are lower level, HTTP (and various other protocols) are higher level. Just because something is sent “all in one go” with HTTP, doesn’t mean it won’t be split into multiple lower level packets based on the size of the message. This seems like some “tech talk” that is there in an attempt to impress people…. and it seems to be working….
While I appreciate details as much as the next person with a “real tech fetish”, I’d like to see a little less talk and A LOT more action. I’m not overly concerned with how information is distributed, just do it reliably and securely (as needed). Don’t over-promise and under-deliver, that just pisses people off. You’re bumping up expectations and lately all we get are constant let-downs. We’ll need to see some positive results from this (and every other change that promises to fix the world) otherwise, attitudes will continue to spoil. So, let’s see how well this big change works, and how little we notice.
December 21st, 2006 at 2:42 PM
Wow, sounds fantastic! Really hope it works out! Thanks for telling us what you’re doing, and also, if it does happen to go wrong (*crosses all claws to forbid*), please keep us informed so that we can be patient and support you instead of moaning.
I have no idea what kept LL from such developmental principles in the first place (money? staffing levels? expertise?) but I’m hugely pleased to hear that you’re practising them now. This sounds like nuts and bolts improvement, system overhauling right where it’s needed and has so long been clamoured for.
Keep up the good work
December 21st, 2006 at 2:55 PM
Nifty Protocol
Gives us better SL Now
Many Messages Flow!
December 21st, 2006 at 3:01 PM
YAY! I am very happy about the changes that are on their way!
December 21st, 2006 at 3:15 PM
So tell me. When does this hit Aditi ?
December 21st, 2006 at 3:20 PM
Holy crap :O
Well, congrats! That’s sheer brilliance
December 21st, 2006 at 3:30 PM
Sounds great Zero, but development better hurry the heck up! What are you going to do if all 2 million residents decide to be online at the same time? Will you at LL be ready? Will any of us? :}
December 21st, 2006 at 3:47 PM
I’m not sure anyone fully understands the consequences of changing the underlying communication of SL.
UDP- Object failed to rez because packet was dropped
TCP object failed to rez because client timed out
Wouldn’t packets arriving late cause more of the “rubberbanding” effect we all experience when building? Are we better off dropping a packet altogether. I assume this is the reason UDP was used in the first place.
Also, is constantly pushing updates going to affect our performance? Are we better off spending 10 min reloading the program every 2 weeks?
What does this do to our vulnerability to viruses like hacks as updates are now being fed to us live? What happens if someone finds a way to spoof a LL update server? How many people let Microsoft update your computer automatically?
I don’t mean to troll, but this a little like listen too a surgeon explaining how he’s just doing a small operation to replace your spine. Easy for him to do, however, the patient may find it a little harder to walk.
December 21st, 2006 at 3:55 PM
Well these are good news Lindens, in particular the “parcel-update-no-one-will-even-notice”, but hey: This was just today
I am 50% of the time invisible
I am 20% of the time without any money on my account
I am 80% of the time seing my lines from 2 minutes ago show up on IM
I am 60% of the time not allowed to TP where I want
I am 50% of the time showing up on a different “previous” location
I am 25% of the time with a shoe up my bum or my hair on my legs
I am 20% of the time talking to thin air (soft crashes)
I am 60% of the time seing things disappear from owners inventories, as in “Hey! Where did the Tango balls went?”
I am 90% of the time PISSED OFF
I am 100% behind you
So please, I know SL is LindenLabs baby, but it is our baby too, so please let us enjoy it. Create a parallell world where you can test drive all changes with hand picked volunteers and add them to the REAL SL only when fully functional for at least a month.
Everybody likes Airbags, ABS and Chrome Wide wheels, but if the Jag is in the garage all the time… Where is the driving pleasure.
Please fix the world and let us enjoy a bit more then 15 minutes without problems.
Thanks guys.
December 21st, 2006 at 4:05 PM
Somehow, I don’t think either the issues over the last month, nor what may be coming when you do this switch qualify as “barely notice”. :-/
Might want to check your GoF book a bit more closely; Capabilities are more a Design “Concept”, rather than a Design “Pattern”.
+1 on your Buzzword Bingo score for the effort, though.
December 21st, 2006 at 4:12 PM
“I am 25% of the time with a shoe up my bum or my hair on my legs”
*SNORT* GAHAHAHAHA! Good one, Alma.
Oh, one other thing: Capabilities are a well-over 20-year-old design concept, whose only remote claim to widespread “successful” use is BROWSER COOKIES! Yes, the same things which common XSS attacks use to get your “capabilities” and use them to order crap from an e-tailer on your credit account. Yes, I know; they are so great. :-/
There’s a reason why the capabilities-vs-acls debate hasn’t subsided yet (even after 20 years)… because neither solution is a complete one, nor the best in a majority of situations.
December 21st, 2006 at 4:16 PM
Very good news ! I remain cautiously optimistic, I couldnt agree more.
You guys really dont have a great track record with your updates, however i agree from a tech point of view, that this is a great step forward in the right direction and opens doors for all sorts of other CAPS especially user programmable. (Mind the hackers/spammers/griefers though - this opens doors for them)
I really hope this works ou well, although im sure we will NOTICE the implimentation *grins*
oh and i assume this is tied to HAVOC 2 being around the corner?
Kind regards guys keep up postigns of this kind, it lets the whingers know that you guys are working on improving our world.
Merry Christmas
Josh
December 21st, 2006 at 4:18 PM
Hope it doesn’t make border crossing too much worse in those cases where it’s projected to be slower. It’s already a gamble today.
December 21st, 2006 at 4:21 PM
Great news! Keep up the good work Lindens. I enjoy the game immensely.
IMHO, after listening to the podcast on IInnovate with Philip Rosedale and the answer to the question poised by Andy Grove (Intel CEO), and then reading Cory Lindens Town Hall transcript, it seems Linden Labs is headed in a direction, in terms of R&D, that will perhaps revolutionize the way people can interact with the http://WWW. Exciting times we live in.
December 21st, 2006 at 4:46 PM
Well, you all seem so excited about this, that I’m excited too. But I won’t pretend to know what’s going on.
However - at any point here are we going to have less lag? It’s taking me 5 minutes to walk from my couch to the door and well… sometimes I do have a need to answer the door.
But I’m still addicted to SL. Lag, crashes, shoes up my butt, and everything.
December 21st, 2006 at 4:47 PM
This last technique is notable: It represents an engineering approach to designing with the goal of not taking the grid down. We will be building more new functionality with this kind of “live fallback”. And how do we know the fallback works? The “sim-caps” code has been deployed for about a month - we’ve just been using the live configuration to keep it turned off!
Thats a pretty bold statement considering the facts that coincidently for about the last month we have probably had the biggest series of bugs,crashes,performance and sundry problems since SL started. Or are you trying to say that we have actually been fed these issues as part of your grand planned testing of your new toy ?
So all we can say is if the last months SL problems can be described as barely noticeable (with it switched off ) , show of hands please for that .. you may get away with it .
I would love to give such “innovation” a thumbs up , but you really have a long way to go with your recent history to convince many people I suspect that this is not firstly going to be a long and painful series of mismanaged releases to stabilise . Worse , as you have as far as I can see NEVER admitted to having a flawed approach to releases or agreed to change strategy. Instead we find you mooting technology ( yes on face value very very welcome ) pro-actively giving you a way to continue the way you have been going , continual ongoing beta , nay alpha, testing on your customer base because now you think you can fix them in situ.
Havent you got it yet .. we want stability first and foremost .. then features. The idea of SL techs constantly fiddly with settings and switching things on and off live fills me with dread.
I really do hope I am sooo wrong about this ..
December 21st, 2006 at 5:06 PM
Some seem to be taking sides with a protocol here (UDP or TCP.)
It important to know the advantages and disadvantages of each, TCP may be reliable and have things such as flow control and other fancy stuff. But them things also make it slow.
UDP is simple, it leaves everything such as reliability to the application layer, I’m sure any of you have looked in the viewer’s debug logs and seen messages such as “recieved nnn expecting nnn” and then later on a load of resent messages. As UDP is simple it’s also fast, that I think it one of the main things, speed.
So HTTP would be good for things like textures, but not everything like some seem to suggest.
Also, (though it’s been a while since I saw the RFC,) in HTTP/1.1 you can have persistant connections so you can send a number of requests down the same connection, the idea being to help reduce the overhead creating TCP connections causes.
December 21st, 2006 at 5:32 PM
Awsome news!! Please keep us updated on this!! , but what about lost items in inventory?
Wolfhaven Teleportation HUD v2.0
Rikhad’s Watch
Personal security HUD v3.31
delicious kiss/hugger 1.1 F W HEELS
Rikhad’s Cock
Xcite! Hud Control Panal
Xcite! Lips
High Black Stitch Boot (R)
High Black Stitch Boot (L)
When i wanna wear these objects I get the message:
“Unable to create requested object. Object is missing from database”
December 21st, 2006 at 6:27 PM
We shall see, we shall see.
December 21st, 2006 at 6:58 PM
I’m still a bit skeptic. It looks good on black and white, but LL has been losing my trust and patience over the past while.
Is it just me, or does it seem like LL is coming out with flashy new things to avoid the dogs? When a bridge can collapse at any moment, you don’t add more weight to the top until it’s been reinforced, right? Similar to a comment made earlier, it’s like having an awesome sports car, but with four flat tires.
December 21st, 2006 at 7:11 PM
Great news! This and the town hall have really restored my faith in your efforts. I was just ignorant of what you were doing with the architecture. I really like the direction you’re headed.
December 21st, 2006 at 7:34 PM
Rikhad, I believe the database wants you to be celibate, if it stole your johnson. (ROTFLOL)
December 21st, 2006 at 7:40 PM
IF you did it right. That’s a big if.
It’s getting to the point where when y’all fix one thing, 5 more break. Please please PLEASE make sure it works before you release it - I don’t want anymore of SL to break.
December 21st, 2006 at 8:01 PM
I agree with Angela Glitter and Altwolf, this sounds like more complicating of the plumbing for no reason other than LL has painted themselves into a corner. Don’t be silently forcing new code into my computer thank you very much. Do your little test grid thing and have us download new updates. I wonder if their new high power advertizers are demanding this?
December 21st, 2006 at 8:31 PM
A real *techie* noticed the changes but it is a good step in the right direction. I understand what is being done by the Linden Lab to improve SL. Kudos to the Linden Lab! I must confess that I should report bugs instead of overlooking them. Thanks for your efforts LL in making SL better. All at the Linden Lab gets an ‘A++’. I hope all there at the LL gets an extra calendar years bonus. Btw, excellent Town Hall meeting, content that is. I think it would be nice to hold separate Town Hall meeting for specific audiences(ie; Technical, Owners, & General Residents). You guys and gals still ROCK
December 21st, 2006 at 8:41 PM
First, an apology - I should have made it more clear that this was a techno-geek heavy post, not for the faint of heart. Ditto many of the comments and my replies here.
Second, thank you for all the kind words on my post. If you goal was to give me enough positive reinforcement that I’d blog again…. you’ve succeeded. I’ll try to blog at least once a month on technical issues like this. I’m really excited that so many of you wrote to say how much you liked reading this sort of post.
Beta Grid: These changes will be available on the beta grid in a few days. Stay tuned to this blog. Several of you asked not be “experimented on”, and for us to “make sure it works” — That is what the beta grid is for, but it only works if you come try it out.
Helping teleport: This will make one aspect of teleport and region crossing more reliable. That aspect will likely be a little slower than the best case now, but much faster than the really slow cases. However, please note that there are many steps to teleport and current problems often lie elsewhere.
Accesses to sim content: This is really just the first step. And while it puts much of the infrastructure in place to enable things like external, authenticated access to sim content, we will be concentrating our use of these technologies on…
…Updates, downloads and downtime: Yes, this is the first step in making grid upgrades happen more smoothly and without down time, and letting you choose when (and if) to download updated viewers, as the old ones will continue work with the new grid. But, we have a few more steps to go…
Updates: Most of my comments were about rolling out updates to the simulator code running on our servers. These changes will also enable us to offer smaller, incremental updates to the viewer - but when we get to that point you’ll never have those forced on you - you’ll be able to take them when you want.
Using HTTP proxy caches for textures: Outside of Linden Lab’s networks, no. Access to textures still requires an authenticated, secure channel, and HTTP doesn’t allow caching in those cases (since the cache itself isn’t authenticated). Internal to Linden Lab’s networks, yes. And we do, but this work should allow us to do so even more.
Primcap/Termcap: I can only aspire to designing something as long lived durable as the termcap database!
UDP/TCP: Indeed, as others have pointed out, each has their use. The bulk of our data turns out to be best suited for TCP, and those are the messages that will be switching to this new system. Some of the data is best suited to UDP, and those messages will be staying with the existing system.
Currently switched off: I know there have been many problems in the last month. We have been very forthcoming in pointing out where the problems have been. The vast majority have been database and search related. Internally, no problem in the last month has been attributable to the switched off sim-caps code. So, I’m very confident about that aspect of this project.
Snowball fight: I was indeed there from about 12:15 to 1:15.
Testing: I think, as we has been discussed so many times in the past - there is no way to test how a technology is going to react when eightteen thousand users and four thousand regions are all using at once, without getting that many users to test. We have been testing sim-caps internally for almost two months. But nothing will compare to running it on the live grid.
Jabber: While Jabber does make use of XML and TCP, that is about where the similarity ends. So, alas, no, this effort won’t help in the short term getting SL’s IM and Jabber to work together.
And I’m *SO* making LLSD sugarcubes in world….
December 21st, 2006 at 8:45 PM
guess what it can be noticed. Cannot teleport money is missing and sl is nothing but a headache. Keep up the good work folks.
December 21st, 2006 at 10:15 PM
Well as long as the teleport system isnt as :messed up” (not my choice of words) as it is now then I’d love to see the new update as soon as possible
December 21st, 2006 at 10:39 PM
“Beta Grid: These changes will be available on the beta grid in a few days. Stay tuned to this blog. Several of you asked not be “experimented on”, and for us to “make sure it works” — That is what the beta grid is for, but it only works if you come try it out.”
Enticing people to come test for you only works if you fix the problems that are reported before updates are rolled to live. Sadly, there’s been a bit too much of this happening (reported bugs ignored and rolled to live), which is most discouraging to people who have extensive beta testing and bug-reporting experience. They bust their butts, give you solid repros, even sometimes offer technical solutions to the problems, and the bugs STILL make it to live.
No, not very enticing at all.
If you can’t fix the problems before you have to release, then you back out the changes which cause the problems, and hold them until the next release. Rolling known broken code forward should NEVER be an option; well, at least if you are trying to pass off a quality product.
“Updates: Most of my comments were about rolling out updates to the simulator code running on our servers. These changes will also enable us to offer smaller, incremental updates to the viewer - but when we get to that point you’ll never have those forced on you - you’ll be able to take them when you want.”
Is that a realistic expectation? Wouldn’t that tend to introduce a significant amount of “legacy bloat” into the system, having to support all possible viewer versions beyond a certain release, each with their own quirks and bugs? Wouldn’t that become a nightmare to maintain, as well as introduce inefficiencies which would manifest as ever-increasing overhead lag?
“Testing: I think, as we has been discussed so many times in the past - there is no way to test how a technology is going to react when eightteen thousand users and four thousand regions are all using at once, without getting that many users to test. We have been testing sim-caps internally for almost two months. But nothing will compare to running it on the live grid.”
Back when Lotus released Notes, I recall hearing about them using a load-simulation tool that they used to simulate up to 10,000 simultaneous users against a Notes/Domino server/cluster. I forget what they called it now, but is it not possible to make such a loading tool which could stress your preview/beta grid at least, so that load-related issues could be investigated more thoroughly? Might be a wise investment.
December 21st, 2006 at 11:12 PM
To Zero / I understood you just fine / excellent job at describing a complex subject / keep up the good work :_)
December 22nd, 2006 at 1:03 AM
It seems pretty obvious they’ll choose something that can apply critical updates and block further loading of older clients when it’s called for. Likely they’ll segment the client into modules that can be updated more or less individually. Perhaps we may even see some of these modules open sourced.
When a module or viewer gets too out of date, it’s deprecated. This is common. For the most part, you only have to bug the user once to install the update — but this will segment the viewer into smaller, easier to download and faster to apply chunks, instead of the monolithic 30MB viewer download we have every time now.
Here’s an example of how this process might work.
The viewer is split into modules, one of which, say, is the Mozilla rendering engine for F1 help, and web profiles. Now, say a new vulnerability’s found in Mozilla Firefox. LL takes the patch from upstream, checks it, and applies it to their tree, and produces a new Mozilla Rendering Engine module version for SL. Now, that single module can be downloaded and perhaps even loaded on the fly, while you’re still in world, and the whole download is perhaps 4MB if the whole package is transferred, and perhaps a few hundred KB if a common protocol like rsync was used that only transfers the differences in the file from the last received version.
Zero, please do keep posting! In fact, I’d be appreicative if you perhaps blogged a bit more than once a month, remember, wordpress has categories and tags! There’s plenty of us geeks who understand this stuff and can do the work of explaining it to the rest of the crowd
December 22nd, 2006 at 2:11 AM
All i know is - i keep reading about changes and new features, while the old ones still aren’t working. It never stops and just gets worse all the time. ***If Lindens would really have an ear inworld***, then they would have noticed that most of the people couldnt care less about new changes that just cause additional problems and therefore want the old bugs out, which is not happening. Might be a nice job for the programmers to take care of their past mistakes, but tat is so uncool, right?
Whenever the systems runs halfways smooth for 1 week the next major crapdate happens…. so 80% of the time it has been a pain during the last year.
I know this is the internet, and in the internet technology advances rules over functionality…
December 22nd, 2006 at 2:26 AM
I agree with you in some ways, Marmela. But, from the sound of things, this is going to work to fix a lot of bugs. Not all bugs are fixed by patching a small bit of code; sometimes you have to rewrite the whole thing, starting from a different angle.
This is very promising, and I hope you guys the best of luck with it! One thing that could come from it is that, instead of doing a huge rollout every couple of weeks with a bunch of features, they can be rolled out one by one, given a few days to make sure they’re ok, then roll out another. Baby steps, instead of jumping to the moon every two weeks, and often falling short…
December 22nd, 2006 at 2:30 AM
Sounds good. Develop it well and test it VERY exhaustively.
December 22nd, 2006 at 2:55 AM
Quote
—
Beta Grid: These changes will be available on the beta grid in a few days. Stay tuned to this blog. Several of you asked not be “experimented on”, and for us to “make sure it works” — That is what the beta grid is for, but it only works if you come try it out.
—
Perhaps LL should offer some kind of incentive for users to spend days on the beta grid. Most of us are of the opinion that Second Life is a finished product, many make thousands of dollars a day with virtual businesses.
I don’t see “beta” anywhere on the site and its bad enough that we report bugs daily, but we are NOT beta testers. We are paying to play (many of us) and we are within our rights to expect a solid product.
If that means finding a way to to pay beta testers then most would probably see it as a good move.
December 22nd, 2006 at 3:11 AM
..”
>Outside of Linden Lab’s networks, no. Access to textures still requires an authenticated, secure >channel, and HTTP doesn’t allow caching in those cases (since the cache itself isn’t >authenticated).
“..
I just want to point out certain potential problems:
+Very Very wise of you to do changes feature by feature while switching to http(s) It removes unpredicted complications impact. Possible complications:
-As you will move towards standard http(s) you might notice more “standard” devices picking up your traffic and making small “corrections” to them. These small changes in your trafffic are hard to catch as traffic will look normal but some http header request fields will be changed.
TRANSPARENT PROXIES:
Example cisco transparent proxy migth generate you other error http server messages than expected by SL client.
Some companies with delpoyed transparent proxy (SNAT ISA2006) will replace your certificate with their approval GATEWAY certificate in order to sniff into your SSL payloads to scann it against maliciouse staff… this would certainly break SL functionality as SL client will not be able to decrypt first part of SSL handshake when public SL key is used.
Some Transparent proxies holds data in order to better force caching system or to scan them.. forget http rfc recomendation about forcing proxies to not apply cache on certain payloads.. generally there are too many ways to define it whcih in real life results in no 1 way of protecting againstit. Best way of protection, from my experience, is UNIQUE URL REQUEST.
Personal Firewalls or Virus scanners:
Modifying cookies: Some proxies try to filter cookies in clever way(for sake of security) …this is like security screening with fuzzy approach … lots of unhandled web service situations.
This is probably true also for some personal firewalls or even AV scanners.. for example Kaspersky Antivirus scans http traffic even it is not sold as firewall (they have another package for that).
Upperlayer mind stack dump DONE;) been working on gatewayes and firewalls for 8 y. now. so i felt urge to say something
my distance view(as iam not seeing deeply enough even after reading your blog entry)suggestions:
tcp/ip timeout less then 1-2 min, use tcp keep alive(very handy in your case) but dont rely 100% on it.
http request/replies headers under 16-32K! (max).
Certificate Revocation capabilities TESTED before deployment.
Put parameters in URL(max 2048chars in IE7) as much as you can. This with added uniqueID functions as best proxy caching protection ever.
December 22nd, 2006 at 3:26 AM
Forgot important part:
Proxies(also trasparent) deployed by lot ISPs do following:
When
web server crashes
or web service request fails
or any router on the way which is state keeping routed tcp/ip connections is restarted
http tcp/ip connection is finished with RST packet
However proxy server sends FIN packet instead of RST packet.
Unfortunatelly this in terms of HTTP communication means gratefull ending and already received data are processed by web service client further which results sometimes in crashes while processing corrupted data.
Lesson learned: Validate your HTTP received trafic yourself dont rely on HTTP ways of validating data.
December 22nd, 2006 at 3:36 AM
I am 200% with Alma Fushikizoh and what she/he said. other then one thing I am 1000% stuck in lag from H*LL. I wish some one would work on that.
December 22nd, 2006 at 5:39 AM
Am I the only one who sees the irony in all of the posts that state, in essense: “SL is COMPLETELY BROKEN and you’d better not do ANYTHING to fix it or we’ll complain even more!” (Insert as many extra exclamations and numeral ones as desired.)
Get a grip, people. This will fix some of the problems you’re complaining about. It won’t fix all of them, but if you’re looking for the one simple thing that will fix all of SL’s problems, there isn’t one. This change is laying a good foundation for some solid future work that should make SL more reliable in time.
Software engineering is not a simple discipline - there are always tradeoffs to deal with, and no matter how hard you try, there is always one more bug. Anyone who’s done serious software development knows that. Anyone who hasn’t done serious software development has no business flaming the guys who are trying hard to fix the problems.
December 22nd, 2006 at 5:56 AM
Zero - good job! While you’re at it, please check the lively mix of secure and non-secure items on the SecondLife website(s).
Just a thought - anticipating a lot less downtime for updates (one day), maybe -for a start- the “downtime for scheduled update” could be replaced by “uptime for scheduled live test”. The main grid open, the kids in the pool, playing with all the new features, but everyone well aware that it is a live test to be taken down at noon PST for evaluation.
Wandarich - we *are* continously being “experimented on”; not as much in SL as it happens in RL. Politics, medications, marketing, etc. etc. Pretty much every aspect of life (RL) involves mass experimenting on commoners/consumers.
Business consideration: “I happily lose one thousand today in order to earn 10 thousand tomorrow.”
December 22nd, 2006 at 7:19 AM
I for one, welcome our big yet barely noticed overlords
December 22nd, 2006 at 8:43 AM
Awesome! so does this mean that SL can use the same search functions as say Ebay now?
December 22nd, 2006 at 9:45 AM
Huh?
December 22nd, 2006 at 10:48 AM
with live help gone there is no longer a way to let the lindens know there is something bad wrong going on. but there is something up with the grid right now. maybe need to just shut back off whatever that bs is you just turned on today. let us have a linden induced grid outage for the holidays please.
December 22nd, 2006 at 11:06 AM
I wonder if this might be the cause for my being unable to teleport right now?