Woman with VR goggles

Unity ARFoundation: Loading XRReferenceImageLibrary For ARCore At Runtime

In the Unity forum, Lorenzo Valente posted an approach to dynamically loading ARCore imgdb files at runtime. His apporach used Reflection to call internal members, which (in his own word) “could make you cry”. So taking it from there (and I can’t emphasise enough how valuable his findings were for me), I explored the ARFoundation source code and found a new solution which I want to share with you.

Basically, the solution contains the following steps:

  1. Register a custom XRImageTrackingSubsystem which contains almost the same code as ARCoreImageTrackingProvider (except that it loads the database from a remote location).
  2. Disable the ARTrackedImageManager in the Editor, so it doesn’t start loading the image database directly at start.
  3. In my script upon awake, deserialize a new XRReferenceImageLibrary with my new reference images (but that’s just some meta info, not the imgdb file itself, yet).
  4. Enable the ARTrackedImageManager in my script, which triggers loading of the database.
  5. My custom XRImageTrackingSubsystem gets called and it in turn loads the imgdb file from my server.

Now, for the steps in more detail:

Register a custom XRImageTrackingSubsystem

The included XRImageTrackingSubsystem for ARCore is called ARCoreImageTrackingProvider and can be found in \Library\PackageCache\com.unity.xr.arcore@2.1.0-preview.5\Runtime\ARCoreImageTrackingProvider.cs. Copy this file to you own project and rename it. You only need some minor adjustments.First, change the point in time the provider gets registered. We need our custom provider to get registered before the built-in provider beacuse the only the first one gets used. So change this lineā€¦

[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
static void RegisterDescriptor()
{
     // ... 
}

… to this line…

[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterAssembliesLoaded)]
static void RegisterDescriptor()
{
     // ...
}

Next, change the GetPathForLibrary method so it returns the URL of your imgdb file on the server. Here’s the original code:

internal static string GetPathForLibrary(XRReferenceImageLibrary library)
{
     if (library == null)
          throw new ArgumentNullException("library");

     return Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
}

And here’s my code:

internal static string GetPathForLibrary(XRReferenceImageLibrary library)
{
    if (library == null)
        throw new ArgumentNullException("library");

    return library is XRReferenceImageLibraryWithDynamicLoading libraryWithDynamicLoading && !string.IsNullOrWhiteSpace(libraryWithDynamicLoading.Url)
        ? libraryWithDynamicLoading.Url
        : Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
}

You may notice that I have introducted a type called XRReferenceImageLibraryWithDynamicLoading. This is just a subclass of XRReferenceImageLibrary that contains an additional property for supplying the URL. This is my aproach to manage URLs, your’s might look differently.

public class XRReferenceImageLibraryWithDynamicLoading : XRReferenceImageLibrary
{
     public string Url { get; set; }
}

Disable the ARTrackedImageManager in the Editor

Well, duh…

Disabled AR Tracked Image Manager component in the Inspector of the Unity editor.
Disabled AR Tracked Image Manager component in the Inspector of the Unity editor.

Deserialize a new XRReferenceImageLibrary / Enable the ARTrackedImageManager
Your image library should match your image database.

    public void Awake()
{
     this.trackedImageManager = this.GetComponent<ARTrackedImageManager>();

#if !UNITY_EDITOR
     var json = @"{
          ""m_GuidLow"": 5140542808452585354,
          ""m_GuidHigh"": 2180237115512313227,
          ""Url"": ""http://172.20.10.5:6789/myimages.imgdb"",
          ""m_Images"": [{
               ""m_SerializedGuid"": {
                    ""m_GuidLow"": 5679415376573207540,
                    ""m_GuidHigh"": 6089316183866679477
               },
               ""m_SpecifySize"": false,
               ""m_Name"": ""poster-of-cool-movie""
          }]
    }";
    var library = new XRReferenceImageLibraryWithDynamicLoading();
    JsonUtility.FromJsonOverwrite(json, library);
    this.trackedImageManager.referenceLibrary = library;
#endif
    this.trackedImageManager.enabled = true;
}

Of course I’d normally get the JSON from a server as well, but for testing purposes I inlined it here. Notice the #if !UNITY_EDITOR preprocessor directive. If you omit that, this code will be called in the editor. Since I don’t want any dynamic loading and server calls to be performed in the editor, I just disabled it that way. The last line in the method above enables the ARTrackedImageManager which triggeres all the internal loading and with it our next step.

Load the imgdb file from my server

Back in my custom XRImageTrackingSubsystem, since I provided the URL of a remote server instead of a path to a file on the device, the built-in code is already capable of handling it. I’ll show the code here, but note that this wasn’t code I added, but code that was already included in the original ARCoreImageTrackingProvider. The only modification I made was inside the GetPathForLibrary method I described above.

public unsafe override XRReferenceImageLibrary imageLibrary
{
    set
    {
        if (value == null)
        {
            UnityARCore_imageTracking_setDatabase(null, 0, null, 0);
        }
        else
        {
            using (var uwr = new UnityWebRequest(GetPathForLibrary(value)))
            {
                uwr.downloadHandler = new DownloadHandlerBuffer();
                uwr.disposeDownloadHandlerOnDispose = true;
                uwr.SendWebRequest();
                while (!uwr.isDone) {}

                byte[] libraryBlob = uwr.downloadHandler.data;
                if (libraryBlob == null || libraryBlob.Length == 0)
                {
                    throw new InvalidOperationException(string.Format(
                        "Failed to load image library '{0}' - file was empty!", value.name));
                }

                var guids = new NativeArray<Guid>(value.count, Allocator.Temp);
                try
                {
                    for (int i = 0; i < value.count; ++i)
                    {
                        guids[i] = value[i].guid;
                    }

                    fixed (byte* blob = libraryBlob)
                    {
                        UnityARCore_imageTracking_setDatabase(
                            blob,
                            libraryBlob.Length,
                            guids.GetUnsafePtr(),
                            guids.Length);
                    }
                }
                finally
                {
                    guids.Dispose();
                }
            }
        }
    }
}

Freelance full-stack .NET and JS developer and architect. Located near Cologne, Germany.

Leave a Reply

Your email address will not be published. Required fields are marked *