独家优惠奖金 100% 高达 1 BTC + 180 免费旋转

Building Multiplatform Mobile Apps With GraalVM Native Image

An in-depth, step-by-step guide to building a voice dictation app

It is a simple voice dictation app. The main points in this demo are incorporating platform services with the domain model. We’ll first acquire speech recognition text from iOS and query a Natural Language Processing (NLP) server to determine whether the captured sentence was a question. And the resulting domain model data will be delivered to ViewModel for further transformation.

The iOS app will demonstrate how to interact with the core component’s C interface. This involves consuming flow/channel published by the core component and exporting platform service(s) through C function pointers.

This demo app was made with GraalVM Community Edition (CE). GraalVM Enterprise Edition (EE) is available at a cost, but it is unnecessary. You will get access to a richer feature set, most notably better GC algorithms, if you elect to go with EE instead of CE.

This was tested on GraalVM version 22.1.0 or lower. The higher version is not working at the moment due to modules not importing correctly. I will update you as soon as that is working.

The architecture here is MVVM with a slight adjustment. Of particular note is that ViewModel is split between the host platform and platform agnostic ViewModel. It is done this way to share non-platform-specific viewmodel logic could be shared.

The domain model handles all business logic and is the source of truth. It handles much of the concurrency to keep data consistent across observations. It is designed in actor paradigm, so its updates are published in an orderly fashion.

All the business logic is represented as pure functions, therefore it’s easy to test.

Let’s take a look at an example of this type of transformation. Here we have a StateFlow for listening to the state of the speech recognizer.

We then transform this state for the Android ViewModel.

And on the corresponding ViewModel on Swift.

Well, you may be wondering how a Kotlin StateFlow can be observed and passed to a mapper function in Swift. To be sure, there is quite a bit of work to be done in between, so let’s take a deeper look.

Isolate is a mini container of Java runtime. It maintains the execution stack and heap of your Java/Kotlin code compiled to the target platform.

A detailed article about Isolate is here: Isolates and Compressed References: More Flexible and Efficient Memory Management via GraalVM | by Christian Wimmer | GraalVM | Medium

An isolate must be created first and then used in all native method invocations. The isolate is bound to a specified thread on the creation, and native invocation must include the thread reference. If native invocation needs to run on a different thread, the isolate must be detached and then reattached to the target thread before invocation.

Isolate must be destroyed manually and will not be reclaimed by the host OS automatically

Isolate C API:

Isolate C API can be used in Swift directly.

In the demo app, we’ll create a dedicated Swift thread and attach an isolate for the duration of the screen rather than create and then destroy each function invocation.

You’ll first need to create a C header file for declarations. Then import the header file into Java.

All methods we wish to expose as C functions must be static and annotated by @CEntryPoint in a class annotated by @CContext. @CContext is analogous to import <header> directive in C.

Lastly, you must accept IsolateThread as an argument in @CEntryPoint function.

More complex data types like structs are passed and returned as pointers. PointerBase is equivalent to a pointer in C.

The above struct is defined in the C header first, then imported into Java. This is the declaration in the C header file as follows:

We can only work with types defined in a header file other than primitives. Boolean requires a standard header file stdbool.h. The string is not a standard type in C. It is a pointer to char and terminated by null. It also means no Generics at all.

Note that in C, evaluation is top-down. Therefore, you cannot reference things backward. Remember that order of declaration matters in C.

We can also import/export C function pointers. For a function pointer declaration in the C header:

In Java, the above is imported as shown:

UnmanagedMemory package provided heap allocation and deallocation of C types.

Allocated memory is released with free, just like in C.

N.B. Native Image C API will most likely be deprecated and replaced by Project Panama Foreign Function and Memory API. Project Panama is in Java 19 as third preview.

In addition, Swift can represent closures as C functions using @convention(c) annotation.

The above declaration can be initialized as:

Alternatively, we can assign a function with a matching signature:

We can pass these function pointer instances to GraalVM C interface, such as CError example in the previous section.

First, we’ll need to convert collect to the observer pattern in Kotlin/Java.

Then, in Java interop layer, we wire in C callback object.

Here’s how the Swift C interop callback is defined:

In this demo app, we have three projects and five modules.

Kotlin Core module is pure Kotlin/Java module since Java C Interop references it.

The Java C Interop module exposes Kotlin Core functionalities and maps data structures from Kotlin/Java to C.

The iOS app module then consumes C functions and data structures, as well as provides infrastructure services to C interface. In this demo app, iOS Speech Recognition is exported through function pointers.

The Java Agent App module is optional. It is responsible for executing the parts of your code with reflection and generating reflection configuration to be used during AOT compilation. In this case, it is needed for Moshi KotlinJsonAdapterFactory for reflective Json deserialization. You may also write this configuration manually or use the codegen version to eliminate reflections altogether.

The domain model serves as the source of truth for the app. It consists of both data and behavior. DTOs are used to construct the data model portion of the domain model. Behaviors are pure business functions that are very easy to test.

This layer is completely platform and client agnostic. It could be reused to serve any client whether it’s mobile or desktop.

It handles concurrency to keep its state consistent at all times. It follows the actor model along with Kotlin coroutines’ structured concurrency.

The model layer orchestrates the domain model and services, then exports observable states (platform agnostic ViewModel) to clients.

Exception propagation across the C interface is done with Either monad in C. This is my attempt at the implementation:

Also, we can enable -H:+ReportExceptionStackTraces and -H:+GenerateDebugInfo=1 in build.gradle for more detailed crash output.

And in case of a crash with Xcode dropping you into breakpoints in assembly, it is most likely an issue with pointers or memory allocation in your Java C interface code. In these cases, there’s not much we can do but “make intuition, then prove/disprove it” cycle in my experience.

Works with AOT with the provided reflection configuration file in the Resource directory. It has a bigger runtime memory footprint, so in this app, we use iOS URLSession instead.

Used with Ktor for Json deserialization. For AOT, it should be used with Agent to generate reflection configuration.

URLSession needed an extension function to make it run synchronously because it already runs in the coroutines IO dispatcher. Elected to use iOS HTTP client because there’s a significant reduction in memory footprint compared to Ktor for AOT compilation.

Moshi Json mapper was used, and it works well apart from needing a configuration file for reflective DTO instantiations.

No issues whatsoever with Koin.

The only caveat here is that there’s no Dispatchers.main, which is only available in environments with a main event loop such as Android or Swing. You’ll have to use a callback handler through the C interface, which submits closures to the iOS main dispatcher. And example of this is in the “Reactive Streams in C/Swift” section above.

The easiest way to run Agent for reflection configuration generation is to use the GraalVM Native Image plugin for VS Code. It will provide Gradle tasks to attach Agent. You may use this Gradle task command: ./gradlew app:run -P agent. After that, you’ll have to point to the directory where the generated files are located, e.g., app/build/native/agent-output/run

Stanford CoreNLP server is needed to parse the text provided by the speech recognizer. You may download it here:

Simply extract and run this command from where it was extracted to:

Then create file nlpserver.properties under speechcore/src/main/resources/

Rebuild is needed after changing this file.

The source code is available on GitHub:

Add a comment

What is your view?

Send

Related posts:

Common Signs Of Leaky Water Lines In Your Property

Water leaks can be elusive, oftentimes developing behind the walls, floors, and ceilings of our plumbing systems. Leaks can go unnoticed for long periods of time and as a result, cause major damage…