Re: Assets not being found, including “MISSING IMAGE” issues
Wednesday, September 13th, 2006 at 11:14 AM by: Torley[UPDATE - Sep. 13 @ 11:26 AM PDT/SLT] Some have asked if today’s update includes fixes to reduce asset server load. Yes.
Hello, just a reminder Second Life is planned to be down from 7 AM-12 PM PDT/SLT while we upgrade to version 1.12.1.
We know there’s been many concerns, including on the Second Life Answers forum, about the distressing and totally sucky experience of getting “MISSING IMAGE” and errors about stuff in your inventory not being found in the database.
We are aware of the issue with assets not being found, and we are working on fixing it.
There is a fail safe, although it isn’t great. If you get this message, our failsafe should detect it and restore the asset within an hour.
For right now, if you get this message, and then still get it over an hour later please send me the asset id for the item if you can. You can get the asset ID by Right clicking on the item in your inventory and choosing ‘Copy Asset ID’. Then paste into any text editor or email client (ctrl-v on windows). If you can’t get the asset ID (for example, if you don’t have copy permissions on the item) then send the full path in your inventory to the item. For example: My Inventory -> Clothing > Cool Outfit > Bling Necklace.
Please also include your avatar name in the email!
Send those emails directly to me: kelly@lindenlab.com.
Additionally, you may have to log out and back in if this blocks an attachment point. I will investigate that bug further as well.
Sorry again, I know this isn’t a good situation.
Also, further technical details from Kelly:
We are sorry for these issues, I am personally sorry for these issues.
As I posted elsewhere we do have a failsafe in place, but it isn’t pretty. Any item that generates this message should be caught by this fail safe and restored within an hour. If it does not, please email me directly (kelly@lindenlab.com). If you can, include the asset IDs of some items that were not restored within an hour. You can get the asset ID by right clicking on the item and choosing ‘Copy Asset ID’ then pasting into any text editor or email program.
What is happening?
Here is some insight into how our asset server works. First, the message is misleading. This isn’t a database issue, we have an asset system that is entirely separate from the database.On this asset system we store all assets in Second Life. An asset is any uploaded image, sound or animation. Assets are also any created notecard or script or landmark. Objects that are not currently being simulated (they are in your inventory or the inventory of an object or in a notecard) are also assets. There are quite a few assets, and new ones get created at a fairly good rate (I don’t have exact numbers on me).
What we do is we run what is called Garbage Collection on the asset server as a whole. What this does is looks for any asset that is no longer referenced anywhere in our system. This means it isn’t in any ones inventory or any objects inventory, it isn’t referenced in a script or in a notecard, it isn’t part of any object (textures). These assets that aren’t referenced anywhere are orphaned - there is no way to ever find them again and no one is using them now. We move these assets aside and mark them for deletion - but we don’t actually delete them.
Instead we wait for a while to make sure that they really and truly aren’t needed, and that GC didn’t screw up.
When GC screws up we get what we are seeing yesterday and today. Assets that are obviously still needed are getting moved aside. We are working on both restoring all the moved aside assets from this GC run and fixing the GC. This is going slower than we had hoped. While this is running the failsafe mentioned above is still running - any asset that can’t be found will be caught by the failsafe and restored.
Again, I am sorry for the trouble this is causing everyone. Your data is not lost and will be restored, although you are right to expect this not to happen in the first place.
And I’m personally sorry for not blogging this earlier.
September 13th, 2006 at 11:26 AM
Thank you for the detailed post about this issue. It’s shed a lot of light on things.
September 13th, 2006 at 11:40 AM
Thanks a lot and keep up the good work!
September 13th, 2006 at 11:43 AM
Thank you for this! It’s a lot more comforting to know my friends with very specialized shapes and attachments will be able to look come back after a little while and be back in their familiar forms!
September 13th, 2006 at 11:53 AM
So basically in laymans terms.
Stuff gettin moved to the trash that shouldn’t of been tossed (Like a real world thing for example a brand new 44 inch plasma TV), the failsafe goes dumpster diving for these files (or objects) and grabs them back out tossing them to the person they belong to after a period of time.
Makes perfect sense.
Thanks Kelly
September 13th, 2006 at 11:55 AM
Thanks for this post Kelly it’s a great weight off my mind.Don’t take it personally it happens.Regards.
September 13th, 2006 at 12:03 PM
Thanks for the info Kelly. Is the GC the most intelligent item in the Second Life system?
September 13th, 2006 at 12:04 PM
Thanks for the informative post Kelly.
I think a lot of us appreciate the “technical details” you give on this one as it helps to understand why the system is doing what it is doing.
Personally, I welcome more of these high level technical explanations as they are a lot more informative than the “We are aware of the problem and are working on a fix.” Hopefully, other Lindens can follow your example when reporting on issues.
September 13th, 2006 at 12:11 PM
See, that’s why I like SL, Kelly does take it personally. They all do. When something goes wrong I always get the impression there’s someone at LL losing sleep over it, worrying that it’s affecting our fun and trying to think their way through it.
Wish I worked somewhere where people were that committed.
September 13th, 2006 at 12:13 PM
Thanks for posting Torley and Kelly. I just hope that in the future, issues such as this will be posted faster. Communication with residents is too important to slack on as the resideants ARE Second Life. If I were a Linden (no I’m not asking
) my first thoughs when something is obviously wrong is to 1: Fix the issue and 2: Get someone, ANYONE, to at least post a cursory mesage on the blog that the issue is at least known and elaborate on it later. I would help calm the fears of residents as well as save you and the good Live Helpers a lot of grief.
That being said, I alot of peole rant and scream and freak out, myself included. Second Life is like nothing that has been done before and I am sure there are a lot of issues with scalability of not only the platform but the company. I personally apreciate the platform for what it is and there is always room for improvement. Keep plugging away at that improvement and hopefully one day Second Life will become what many of us invision it to be.
September 13th, 2006 at 12:20 PM
So tell me, what are the ramifications if a lot of stuff goes missing and can never be recovered? Are we out of luck and screwed? What will you do if someone says hundreds or thousands of inventory items are missing and they can’t be found on the system? Is LL going to compensate them in any way for the vanished items? Or what about personal creations that are irreplaceable. How do you compensate the time and effort there? Has anyone ever given thought to other backup solutions? Lord knows you wouldn’t trust the average user to keep their own copy on their HD.
September 13th, 2006 at 12:23 PM
Thank you for the clarification, It’s nice to see people who live by, you can delegate authority, you can not delegate responsibility.
September 13th, 2006 at 12:28 PM
Thanks for keeping us secondlifers up dated on the great work.
September 13th, 2006 at 12:36 PM
Great, thanx for the comments–I’ve let Kelly know to have a look here and I enjoy hearing insights from him too!
Also, my job priorities are transitioning so this hopefully means I’ll have more time to be on the blog, meaning yes, more, quicker updates when I can provide them. I’m just one Linden contributing here but I love communicating and letting you guys know what’s up, or relaying info from a fellow Linden that hasn’t been displayed clearly enough (e.g., buried in the forums).
I’m often frothing like a rabid dog to share the news.
September 13th, 2006 at 12:44 PM
I admit, I really don’t understand most of what you are saying. In the past, I have had many assets go missing and probably more missing than I even know of.
In those cases the items in question are COMPLETELY gone from my inventory. Currently, the problem I have had is that the item appears to be in my inventory, but when I click on it, I get that message.
What I am asking is this: What is the difference between something just totally vanishing and something that appears to be in my inventory and is not.
Further, I have at least one item in my inv that went “missing’ several months ago. It’s a shape. When I click on it, I get a message. That item, though it appears to be in my inv, is not, and was never restored. So, my question is: why didn’t the “garbage collector, you speak of, ever restore that item?
I really hope someone responds to these questions.
Thanks for all you all do. I think people sometimes forget that even though SL isn’t perfect, nothing in the world is, especially people. Thus, I wonder what they would think of if anyone came into their RL or SL and began to criticize what they do, constantly.
Being in SL is a choice. Paying money to it is a choice. Anyone is free to choose to be here or not.
Thus, I thank all the Lindens!
But I AM confused about my “lost” objects, lol.
September 13th, 2006 at 12:56 PM
Kelly, your sense of personal responsibility is not only commendable but rare. Your willingness to share it with us is … what shall I say? … completely over the top! Such technical explanations as this are extremely welcome and they really do show that LL is neither “stealing” anybody’s resources nor hiding SL’s blemishes. Thanks for setting a great example!
September 13th, 2006 at 1:07 PM
Like Zoe I run a small business in SL; unlike Zoe it plays no part in my RL finances — it’s just a hobby. And I haven’t had any problems with missing inventory. She has a great store though — my Neko alt goes there a lot — and I’m distressed at the performance, or lack of it, you’re providing her.
YOU Linden’s are the guys who have been trying to attract real businesses, but I think you’ve scaled up too fast, and the system that worked when you had 30,000 subscribers is beginning to crack as it approaches a million. It’s not an adequate response to post an “I’m sorry, sorry, sorry” response. If you’re telling the world you’re a platform, not just a game, you’ve got to stop acting like a bunch of North Beach hippies, start acting like you’re wearing suits, and provide a product that provides enough stability for the Rest of Us.
September 13th, 2006 at 2:46 PM
I had this problem a few days ago and then the item suddenly appeared again in inventory. I just figure it’s another of seemingly endless lil bugs that plague SL at times. But nothing major compared to security issues. I now have my gorgeous white T again that I haven’t taken off since I “found” it. Oh btw what about those 2 hippies that started a lil company called Apple? One of them is still running it last I checked and I see him mostly in turtlenecks.
September 13th, 2006 at 3:32 PM
Thanks Kelly, for your detailed explanation. So that’s what happened to my bed!! I’m lucky insofar as the creator gave me a replacement, but it was an expensive item and it would be good if the original was found, so I could give it back to him. What is the average time span between the object vanishing from ones inventory and your system returning it? Assuming it doesn’t get deleted of course…….
September 13th, 2006 at 3:53 PM
I dang near had a cornoary when I logged in yesterday and one of the items the system couldn’t find was my custom AV skin!! All of a sudden I this pale little nebbish instead of my normal handsome self. Thinking quickly, I tried to don my Siamese Cat AV - to no avail! It too cannot be found. Aiiee! Doomed, I am doomed!
All drama aside, I *hope* that when the current ensnargulations are resolved, that all will be well in SL land, and I thank the Lindens for all their hard work.
September 13th, 2006 at 6:33 PM
The GC system sounds like what I figured you had in place, in general at least. It also I think explains an issue a friend recently ran into. He had rezzed a bunch of no-copy items in a sim and they were out when the grid had an emergency shutdown. When the grid came back up the items were not rezzed in the sim and also not in his inventory. This cost him beteween 3,000 and 5,000 lindens. What appears to have happened from how I know of things working is when the grid shutdown it didn’t take a current snapshot of the sim’s contents so when the grid came back up the sim was ‘rolled back’ to its last snapshot, which was before he put out the items. On the other hand when the grid went down it did take a current snapshot of his inventory which didn’t have the items in it since they were no-copy. What this amounted to was the items becoming ‘lost’ and possibly cleaned up by GC depending on if an object asset is unique per instance of the object. Even if not it indicates a flaw in the system that needs to be addressed as far as grid shutdowns and data snapshots.
September 13th, 2006 at 6:55 PM
Personally, I think Kelly’s explanation deserves a sticky, just because it is such an excellent example how things should be done: the nature of a technical issue is explained in sufficient detail that everyone understands *why* what has happened has happened. I totally agree with Damian Villas, it is incomparably better than just a vague “we know what’s wrong and we are working on it,” and there should be more of this. Kudos to you, Kelly, the time you have spent explaining the issue has *not* been wasted in my book.
September 13th, 2006 at 6:59 PM
I sure hope Kelly sees all of this. Blog posts don’t have stickies per se, Veronica, but we can easily tag and sort them–there are some past articles you may be interested in too, like:
Anatomy of a Second Life Update, also by Kelly
http://blog.secondlife.com/2006/08/02/anatomy-of-a-second-life-update/
Anatomy of a preview, by Phoenix Linden
http://blog.secondlife.com/2006/02/09/anatomy-of-a-preview/
September 13th, 2006 at 7:00 PM
Please give us a tool to export out inventory information including the asset IDs.
We can do this on a frequent basis so our property that goes missing can be recovered.
I too lost 3000L worth of equipment when a sim crashed.
Had i known of the option to have the objects restored i would have manually made a list of the expensive items long ago.
U utility to export this critical and very valuable information would make a huge difference for us all.
Please consider this request.
After all, it should be a simple query to the database. the trick i suspect is getting it into a notecard or exported to disc.
Thank you so much for the detailed information.
September 13th, 2006 at 9:27 PM
The export tool may be a challenge from the importing securely point of view, but it would be good to have a way to save out at least a list.
What I would like to see is an update cycle that is just bug fixes and stability fixes. I appreciate the new features, but I have been seeing consisten load problems and group problems, and asset problems, and friends lists being intermittant. I think a good shoring up the foundations and working out the scaling issues release is really in order - then give us new features.
Just my two cents here.
September 14th, 2006 at 4:01 PM
tOR WE ARE ALL INVISIBLE AND OUR LINDENS ARE ALL GONE ;-(
WHATS THIS ALL ABOUT??????
September 14th, 2006 at 8:16 PM
The only problem with a backup on your own HD would come from the following scenario:
You have several no-copy items. You make a backup. Then you restore those items after a crash. Then the GC system catches up with the event.. no what to do ? One of the items found by the CG was a no-copy item. Which to keep ? The one in your inventory or the one in the GC restore list ?
Also, the possibility of some character trying to crack the cypher codes that would be needed to ensure the local backup can’t be altered would be very high. Someone would be trying to break the encryption needed to make this work as surely as the sun rises in the morning.
Local HD backup *could* be done, but would require encryption that the US government would be very unhappy about seeing loose in the wild. Remember, they have made it illegal for US companies to sell or export many cryptographic tools and algorithms.
Kelly, why not give us a way to snapshot all the asset ID numbers for our inventory and store at least that on our HDs in encrypted form? It would be a way to verify what we had before a crash and might make the CG restore function simpler.
September 16th, 2006 at 11:11 PM
This issue is still happening here to myself and to nearly *everyone* I meet. The inventory problem is still with us and is getting, if anything, worse.
187 people so far, all have problems where items are named in inventory, especially inside boxed items, but have no creator info and, apparently, no content. What worked in the past for these items, click and “open”/”copy to inventory” fails nearly everytime. A *few* boxed items with a “touch” script sometimes work.
wtf ? It’s 3 days later and not a peep when that many people are seeing this ?
September 17th, 2006 at 12:02 AM
^ Ryu, there’s going to be more news on this and other issues that have come up post-1.12.1. A lot of bugs not yet publicly listed have already been identified by Quality Assurance and will be fixed ASAP by our Developers.
Right now it’s the weekend but when I’m back at work I’m going to be doing more regular Community Updates. Promise.
Some new 1.12.1 issues are listed here too:
http://secondlife.com/support/known-issues.php
September 16th, 2007 at 12:54 PM
I know its been about a year since this was posted, but Ann O’Toole was wise enough to let us know about this in her comment under Lotte Linden’s posting “Network connectivity problem” http://blog.secondlife.com/2007/09/15/network-connectivity-problem/#comment-477711.
I am assuming it is so, but Is the GC still used in this way with the asset servers?
A lot of people need to see this so they will stop ranting about their “lost items”.