Unity ARFoundation: Loading XRReferenceImageLibrary For ARCore At Runtime

In the Unity forum, Lorenzo Valente posted an approach to dynamically loading ARCore imgdb (XRReferenceImageLibrary) files at runtime. His approach used Reflection to call internal members, which (in his own words) “could make you cry”. So, taking it from there (and I can’t emphasize enough how valuable his findings were to me), I explored the ARFoundation source code and found a new solution which I want to share with you. This step by step guide will show you how to load XRReferenceImageLibrary For ARCore at Runtime using ARFoundation source code.

Basically, the solution contains the following steps:

Loading XRReferenceImageLibrary For ARCore At Runtime using ARFoundation: The basic steps

1. Firstly, register a custom XRImageTrackingSubsystem which contains almost the same code as ARCoreImageTrackingProvider (except that it loads the database from a remote location).
2. Next, disable the ARTrackedImageManager in the Editor so it doesn’t start loading the image database directly at start.
3. Then, in ‘My script’ upon awake, desterilize a new XRReferenceImageLibrary with my new reference images (but that’s just some meta info, not the imgdb file itself, yet).
4. Finally, enable the ARTrackedImageManager in My script, which triggers loading of the database.
5. As a result, my custom XRImageTrackingSubsystem is called and in turn loads the imgdb file from my server.

Now to clarify the steps in more detail:

Register a custom XRImageTrackingSubsystem

The included XRImageTrackingSubsystem for ARCore is called ARCoreImageTrackingProvider and can be found in \Library\PackageCache\com.unity.xr.arcore@2.1.0-preview.5\Runtime\ARCoreImageTrackingProvider.cs. Firstly, copy this file to your own project and rename it. You only need some minor adjustments. After that, change the point in time that the provider is registered. We need our custom provider to be registered before the built-in provider, because only the first one gets used. So change this line…

[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
static void RegisterDescriptor()
{
     // ... 
}

… to this line…

[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterAssembliesLoaded)]
static void RegisterDescriptor()
{
     // ...
}

Next, change the GetPathForLibrary method so it returns the URL of your imgdb file on the server. Here’s the original code:

internal static string GetPathForLibrary(XRReferenceImageLibrary library)
{
     if (library == null)
          throw new ArgumentNullException("library");

     return Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
}

And here’s my code:

internal static string GetPathForLibrary(XRReferenceImageLibrary library)
{
    if (library == null)
        throw new ArgumentNullException("library");

    return library is XRReferenceImageLibraryWithDynamicLoading libraryWithDynamicLoading && !string.IsNullOrWhiteSpace(libraryWithDynamicLoading.Url)
        ? libraryWithDynamicLoading.Url
        : Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
}

However, you may notice that I have introduced a type called XRReferenceImageLibraryWithDynamicLoading. To clarify, this is just a subclass of XRReferenceImageLibrary that contains an additional property for supplying the URL. This is my approach to manage URLs, your own, for instance, might look differently.

public class XRReferenceImageLibraryWithDynamicLoading : XRReferenceImageLibrary
{
     public string Url { get; set; }
}

Disable the ARTrackedImageManager in the Editor

Well, duh…

Deserialize a new XRReferenceImageLibrary / Enable the ARTrackedImageManager Your image library should match your image database.

    public void Awake()
{
     this.trackedImageManager = this.GetComponent<ARTrackedImageManager>();

#if !UNITY_EDITOR
     var json = @"{
          ""m_GuidLow"": 5140542808452585354,
          ""m_GuidHigh"": 2180237115512313227,
          ""Url"": ""http://172.20.10.5:6789/myimages.imgdb"",
          ""m_Images"": [{
               ""m_SerializedGuid"": {
                    ""m_GuidLow"": 5679415376573207540,
                    ""m_GuidHigh"": 6089316183866679477
               },
               ""m_SpecifySize"": false,
               ""m_Name"": ""poster-of-cool-movie""
          }]
    }";
    var library = new XRReferenceImageLibraryWithDynamicLoading();
    JsonUtility.FromJsonOverwrite(json, library);
    this.trackedImageManager.referenceLibrary = library;
#endif
    this.trackedImageManager.enabled = true;
}

Of course I’d normally get the JSON from a server as well, but for testing purposes I inlined it here. Notice the #if !UNITY_EDITOR preprocessor directive. If you omit that, as a result, this code will be called in the editor. Because I don’t want any dynamic loading and server calls to be performed in the editor, I just disabled it that way. The last line in the method above enables the ARTrackedImageManager which triggers all the internal loading and with it our next step.

Load the imgdb file from my server

Back in my custom XRImageTrackingSubsystem, since I provided the URL of a remote server instead of a path to a file on the device, the built-in code is already capable of handling it. I’ll show the code below. Please note that this wasn’t code I added, but code that was already included in the original ARCoreImageTrackingProvider. To clarify, the only modification I made was inside the GetPathForLibrary method I described above.

public unsafe override XRReferenceImageLibrary imageLibrary
{
    set
    {
        if (value == null)
        {
            UnityARCore_imageTracking_setDatabase(null, 0, null, 0);
        }
        else
        {
            using (var uwr = new UnityWebRequest(GetPathForLibrary(value)))
            {
                uwr.downloadHandler = new DownloadHandlerBuffer();
                uwr.disposeDownloadHandlerOnDispose = true;
                uwr.SendWebRequest();
                while (!uwr.isDone) {}

                byte[] libraryBlob = uwr.downloadHandler.data;
                if (libraryBlob == null || libraryBlob.Length == 0)
                {
                    throw new InvalidOperationException(string.Format(
                        "Failed to load image library '{0}' - file was empty!", value.name));
                }

                var guids = new NativeArray<Guid>(value.count, Allocator.Temp);
                try
                {
                    for (int i = 0; i < value.count; ++i)
                    {
                        guids[i] = value[i].guid;
                    }

                    fixed (byte* blob = libraryBlob)
                    {
                        UnityARCore_imageTracking_setDatabase(
                            blob,
                            libraryBlob.Length,
                            guids.GetUnsafePtr(),
                            guids.Length);
                    }
                }
                finally
                {
                    guids.Dispose();
                }
            }
        }
    }
}

I wish you successful endeavors following this step by step guide for how to load XRReferenceImageLibrary For ARCore at Runtime using ARFoundation source code.