Tag Archives: AssetImporter

Unity: Reading assetTimestamp’s blazingly fast

UnityIn my previous posts (1 2), I was complaining reading the timestamp of an asset using the “AssetImporter.GetAtPath().assetTimestamp” API can be enormously slow.

In this post, I describe what I came up with to read thousands of timestamps in a few milliseconds only, rather than waiting an hour when using the official Unity API.

Upcoming versions of my Unity plugins feature a cache file that holds asset properties which are expensive to read. Once these properties have been read, they are stored in the cache file, so the next time they are requested, they can be served much faster.

The problem I encountered was to detect if an asset has been modified, to refresh the corresponding cache entry, in an acceptable period. Something like the following pseudo-code with 3000 textures stored on a 7 years old harddrive of mine took about 20 minutes to complete:

The issue, as explained in my previous post, is that “AssetImporter.GetAtPath” loads the asset into memory. Thus, the time the call requires to complete highly depends on the asset filesize itself.

While digging in various Unity related project folders to find a workaround, I came across the file “Library/assetDatabase3”. This file stores timestamps, along other data, of all assets in the project.

The format of assetDatabase3 changed from Unity 3.x to Unity 4 and it’s very likely that it can change in a future release too. Thus, it’s important for the plugin not to break when reading assetDatabase3 fails.

While reading this file, Unity shouldn’t updating it either. Running the code from the main-thread will do the trick, since it blocks the application and Unity won’t start to import new assets during this time.

In Unity 4, assetDatabase3 holds at offset 0x0C the offset where the timestamp table is located. In the following image, it tells us it’s located at offset 00 00 6B 80:

assetDatabase3 in HEX-Editor

Seeking at offset 0x6b80 in the same file, gets us at the at the start of the timestamps table:

assetDatabase3 in HEX-Editor

The timetamp table starts with:

For every element it is:

Timestamps need a some extra bit-magic to shift them into the correct order. The necessary bit-magic is also different for Mac and Windows. The final assetTime is basically the greater value of the two timestamps.

Reading assetDatabase3 and extracting all the assetTimestamps takes only a few milliseconds to complete. Once read, the plugin compares each of the timestamps to the cached data entries. If they are different, the cache entry is assumed to be invalid and will be refreshed. If reading assetDatabase3 fails, the plugin handles it as all cache entries are invalid.

This change improves the startup time of my plugins dramatically. I tested it with several projects and different Unity 4.x versions and it doesn’t really matter how many assets they contain. When opening the plugin, such as Texture Overview, it’s ready in about 0.5-2 seconds always (assuming the cache file is uptodate and assets have not changed/reimported). Most of these projects contain about 3000 textures and 1000 audio clips. One of my artificial test project contains 20000 audio clips. The plugin opens and is ready-to-use in all projects almost instantly.

Unity: AssetImporter.GetAtPath causes alot of file system activity

UnityIn my previous post, I wrote I was optimizing my Texture Overview Unity plugin by caching texture properties (width, height, filesize, mipmap count, etc), that usually require expensive file I/O operations to collect. Once these properties have been collected, the plugin adds them to a persistent cache, so they can be served much faster the next time they are requested.

In order to detect if a cache-entry of a texture turned invalid, the plugin compares the AssetImporter.assetTimestamp to the timestamp of the cache-entry. If they don’t match, the cache-entry became invalid and the plugin has to read and collect the new texture properties, rather than just serving the cached data.

I was certain this solution will make the plugin blazingly fast, but to my suprise, it did not. It did improve the startup time of the plugin, but not as much as I thought it would be. The reason for that is AssetImporter.GetAtPath() can be horribly slow.

While the plugin starts, it calls AssetImporter.GetAtPath(“path”).assetTimestamp for every texture in the project, to detect if the cached data for a particular texture is invalid. During this time, the harddrive is extremely busy.

To try to understand what is happening there, I downloaded Process Monitor to monitor file system activity.

I created a little script that just calls AssetImporter.GetAtPath for “Assets/Textures/AngryBots.png” (inside the AngryBots project) and it turns out this inconsiderable method opens three different files, as Process Monitor reveals:Process Monitor: AssetImporter.GetAtPath()

“AngryBots.png” is the source image file. “AngryBots.png.meta” is in Unity’s own words “a text file for every asset in the Assets directory containing the necessary bookkeeping information required by Unity” (see here) and “7b469925895d5466dbc74ebf68ae2a98” is the imported image file in Unity’s own texture asset format, the filename equals the asset guid you can get by calling AssetDatabase.AssetPathToGUID btw.

I’m wondering why Unity is actually opening all of these files. My understanding was the importer settings are stored in the .meta file, so why does it open the imported texture asset (two times) and why even the source .png file?!

I also noticed getting the importer for a huge texture file is slower than for a small one. I thought this makes no sense at all, but now, since I know Unity opens all of these files, this could actually be true.

Calling AssetImporter.GetAtPath a second time for “AngryBots.png” during the same Unity session, only opens “AngryBots.png” and “AngryBots.png.meta”, but not “7b469925895d5466dbc74ebf68ae2a98” anymore, so it’s much faster this time. I guess Unity loads the imported texture into memory and then it doesn’t need to load it again on the seconds call and this would actually support my theory that AssetImporter.GetAtPath performs worse the bigger the imported texture asset is.

Anyway, I will probably never find out why it is reading all these files and whatnot.

Unfortunalety, this performance issue makes AssetImporter.GetAtPath, to get my hands on the assetTimestamp, rather useless again. There has to be a way to get the assetTimestamp of 10000 assets in less than several minutes.