Unity: Reading assetTimestamp’s blazingly fast

UnityIn my previous posts (1 2), I was complaining reading the timestamp of an asset using the “AssetImporter.GetAtPath().assetTimestamp” API can be enormously slow.

In this post, I describe what I came up with to read thousands of timestamps in a few milliseconds only, rather than waiting an hour when using the official Unity API.

Upcoming versions of my Unity plugins feature a cache file that holds asset properties which are expensive to read. Once these properties have been read, they are stored in the cache file, so the next time they are requested, they can be served much faster.

The problem I encountered was to detect if an asset has been modified, to refresh the corresponding cache entry, in an acceptable period. Something like the following pseudo-code with 3000 textures stored on a 7 years old harddrive of mine took about 20 minutes to complete:

The issue, as explained in my previous post, is that “AssetImporter.GetAtPath” loads the asset into memory. Thus, the time the call requires to complete highly depends on the asset filesize itself.

While digging in various Unity related project folders to find a workaround, I came across the file “Library/assetDatabase3”. This file stores timestamps, along other data, of all assets in the project.

The format of assetDatabase3 changed from Unity 3.x to Unity 4 and it’s very likely that it can change in a future release too. Thus, it’s important for the plugin not to break when reading assetDatabase3 fails.

While reading this file, Unity shouldn’t updating it either. Running the code from the main-thread will do the trick, since it blocks the application and Unity won’t start to import new assets during this time.

In Unity 4, assetDatabase3 holds at offset 0x0C the offset where the timestamp table is located. In the following image, it tells us it’s located at offset 00 00 6B 80:

assetDatabase3 in HEX-Editor

Seeking at offset 0x6b80 in the same file, gets us at the at the start of the timestamps table:

assetDatabase3 in HEX-Editor

The timetamp table starts with:

For every element it is:

Timestamps need a some extra bit-magic to shift them into the correct order. The necessary bit-magic is also different for Mac and Windows. The final assetTime is basically the greater value of the two timestamps.

Reading assetDatabase3 and extracting all the assetTimestamps takes only a few milliseconds to complete. Once read, the plugin compares each of the timestamps to the cached data entries. If they are different, the cache entry is assumed to be invalid and will be refreshed. If reading assetDatabase3 fails, the plugin handles it as all cache entries are invalid.

This change improves the startup time of my plugins dramatically. I tested it with several projects and different Unity 4.x versions and it doesn’t really matter how many assets they contain. When opening the plugin, such as Texture Overview, it’s ready in about 0.5-2 seconds always (assuming the cache file is uptodate and assets have not changed/reimported). Most of these projects contain about 3000 textures and 1000 audio clips. One of my artificial test project contains 20000 audio clips. The plugin opens and is ready-to-use in all projects almost instantly.