Source, binary and backward compatibility rule them all

May 13, 2024

As a software engineer, you’ll inevitably encounter compatibility challenges at some point in your career. These may manifest as source compatibility, binary compatibility, or backward compatibility issues. In this article, we’ll delve into these concepts, exploring what they entail and how best to navigate them.

It’s important to note that the content of this article is tailored to Kotlin users, with a focus on Gradle as the primary build tool.

You can reproduce the issues and problems faced in this article using the provided sample project.

Source Compatibility

Source compatibility is the most straightforward form of compatibility, it refers to the ability of the new version of your code to seamlessly integrate without necessitating any modifications in the client’s codebase.

In other words, the updated version of your code can be recompiled with the latest version of your library without extra changes.

Let’s take the following example.

public interface DishesRepository {
    public fun getDishes(): List<Dish>
}

Suppose you intend to modify this interface to retrieve all vegan dishes. One approach is to introduce a specification argument like this:

public interface DishesRepository {
    public fun getDishes(specification: DishSpecification): List<Dish>
}

Predictably, this adjustment would trigger a compilation error, as indicated below:

e: No value passed for parameter 'specification'

The reason behind this error lies in the fact that the alteration made to the code isn’t source-compatible.

An effortless way to ensure source compatibility is to provide a default value for the new parameter, as demonstrated below:

public interface DishesRepository {
    public fun getDishes(specification: DishSpecification = DishSpecifications.all()): List<Dish>
}

With this modification, the code becomes source-compatible with the previous version, allowing it to compile seamlessly without requiring any modifications from the client’s end.

Binary compatibility

In our earlier example, we discussed source compatibility and how to maintain it when we apply changes to our code. But is that all we need? The answer depends on a few things.

There are two situations to consider:

We depend directly on the sources: If we directly rely on the source code of a library, we usually just need source compatibility. This happens when we’re working on internal projects or using a mono-repo for our code, a general rule of thumb to identify if you are in this case is when you are using dependencies like implementation(project(“:lib”)).
We depend on published libraries: When we depend on a published version of a library, like when we add a line such as implementation(“com.msignoretto:lib:1.0.0”) in the build.gradle.kts file, source compatibility is not enough and we need to think about binary compatibility as well. This happens when we use libraries made by others or when we publish libraries that others or ourselves use.

Let’s deep dive into the second case.

Binary compatibility is stricter than source compatibility. It not only means that the client’s code doesn’t need to change, but it also doesn’t need to be recompiled. It ensures that the new version of our code can work with the old version of the client’s code without the client having to upgrade explicitly.

Let’s explain this with an example:

Imagine we initially publish our library without the dish specification as a jar file called 1.0.0 with the name com.msignoretto:lib:1.0.0. Now, if we shift to using the published version, our dependency in the client’s code changes from implementation(project(“:lib”)) to implementation(“com.msignoretto:lib:1.0.0”).

This change is significant. Now, we depend on the published version of the library’s binary, not directly on its source code.

Considering this shift, let’s see how it affects our code.

Suppose we make the changes related to the dish specification, as described earlier, and publish this as a new version, say 1.1.0.

In this case, we might face two situations:

Your dependency tree contains only the new version
Your dependency tree contains both the old and the new version

Let’s explore both scenarios in detail.

Scenario 1

In this scenario, the client doesn’t need to make any changes to their code. They can simply update the dependency to the new version of the library, resulting in the same outcome as with the source-compatible code.

To visualize the dependency tree, you can execute the following command:

./gradlew :app:dependencies

Replace app with the name of your module.

For the sample project, the dependency tree will appear as follows:

...
+--- project :moduleA
| +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
| \--- com.msignoretto:lib:1.1.0
| \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
\--- project :moduleB
+--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
\--- com.msignoretto:lib:1.1.0 (*)

It’s important to note that the output of the dependencies command doesn’t contain the old version of the library at all.

Scenario 2

The second scenario is indeed more complex. Let’s consider what happens when we update the client to use the new version of the library, while still having a dependency compiled against the old version of the library.

To simulate this scenario, we modify moduleB in the sample project to use the old version, 1.0.0.

> If you want to reproduce this exact scenario checkout the binary-incompatible branch of the sample project here.

Now, your dependency tree displays both the old and the new versions of the library:

+--- project :moduleA
| +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
| \--- com.msignoretto:lib:1.1.0
| \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
\--- project :moduleB
+--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
\--- com.msignoretto:lib:1.0.0 -> 1.1.0 (*)

In this scenario, we notice a difference compared to Scenario 1:

Both the old and new versions are present in the dependency tree.
mobuleB 1.0.0 -> 1.1.0 is a new addition that wasn’t present before.

Let’s put that aside for now and come back to it later.

From the output above, you can observe that the sample project is composed of 2 modules: moduleA and moduleB. Both modules contain identical code and both depend on the lib library but with different versions. While moduleA depends on version 1.1.0, moduleB relies on version 1.0.0.

Both ModuleA and ModuleB contain the following code:

class ModuleA {
    fun run() {
        val dishesRepository = DishRepositoryFactory.create()
        dishesRepository.getDishes().forEach {
            println("${it.name} - ${it.description} - ${it.price}")
        }
    }
}

Let’s try to execute the following function:

fun main() {
    println(">> Module A <<")

    val moduleA = ModuleA()
    moduleA.run()

    println(">> Module B <<")

    val moduleB = ModuleB()
    moduleB.run()
}

Upon running that function, we encountered the following error:

Exception in thread "main" java.lang.NoSuchMethodError: 'java.util.List com.msignoretto.binarycompatibility.lib.DishesRepository.getDishes()'
at com.msignoretto.binarycompatibility.moduleA.ModuleA.run(ModuleA.kt:8)
at com.msignoretto.binarycompatibility.app.MainKt.main(main.kt:10)
at com.msignoretto.binarycompatibility.app.MainKt.main(main.kt)

This error serves as a clear indication that your code is not binary-compatible.

Upon decompiling the bytecode for moduleA, we encounter the following bytecode for invoking the getDishes() method:

INVOKESTATIC com/msignoretto/binarycompatibility/lib/DishesRepository$DefaultImpls.getDishes$default (Lcom/msignoretto/binarycompatibility/lib/DishesRepository;Lcom/msignoretto/binarycompatibility/lib/DishSpecification;ILjava/lang/Object;)Ljava/util/List;

Meanwhile, for moduleB, the bytecode for the invocation is as follows:

INVOKEINTERFACE com/msignoretto/binarycompatibility/lib/DishesRepository.getDishes ()Ljava/util/List; (itf)

For improved readability, let’s decompile the bytecode of both modules into Java, resulting in the following code.

ModuleA:

public final class ModuleA {
   public final void run() {
      DishesRepository dishesRepository = DishRepositoryFactory.INSTANCE.create();
      Iterable $this$forEach$iv = (Iterable)DefaultImpls.getDishes$default(dishesRepository, (DishSpecification)null, 1, (Object)null);
      int $i$f$forEach = false;
      Iterator var4 = $this$forEach$iv.iterator();

      while(var4.hasNext()) {
         Object element$iv = var4.next();
         Dish it = (Dish)element$iv;
         int var7 = false;
         String var8 = it.getName() + " - " + it.getDescription() + " - " + it.getPrice();
         System.out.println(var8);
      }
   }
}

ModuleB:

public final class ModuleB {
   public final void run() {
      DishesRepository dishesRepository = DishRepositoryFactory.INSTANCE.create();
      Iterable $this$forEach$iv = (Iterable)dishesRepository.getDishes();
      int $i$f$forEach = false;
      Iterator var4 = $this$forEach$iv.iterator();

      while(var4.hasNext()) {
         Object element$iv = var4.next();
         Dish it = (Dish)element$iv;
         int var7 = false;
         String var8 = it.getName() + " - " + it.getDescription() + " - " + it.getPrice();
         System.out.println(var8);
      }
   }
}

If you diff those files you will get the following output:

< public final class ModuleA {
---
> public final class ModuleB {
4c4
<       Iterable $this$forEach$iv = (Iterable)DefaultImpls.getDishes$default(dishesRepository, (DishSpecification)null, 1, (Object)null);
---
>       Iterable $this$forEach$iv = (Iterable)dishesRepository.getDishes();

This comparison highlights that moduleA expects to find a method:

DefaultImpls.getDishes$default(dishesRepository, (DishSpecification)null, 1, (Object)null);

while moduleB expects to find

dishesRepository.getDishes();

Hence explains the runtime error faced above.

To understand why and how this occurs, let’s delve into how Gradle resolves dependencies.

Gradle dependency conflict resolution

When employing Gradle as your build tool and facing a dependency conflict, Gradle defaults to a particular conflict resolution strategy. This strategy entails utilizing the latest version of the conflicting dependency.

Let’s examine again Scenario 2 discussed earlier.

Our project has two modules, moduleA and moduleB which are both built together as part of the app. moduleA depends on libversion 1.1.0, while moduleB depends on version 1.0.0.

When we build the project, Gradle identifies the conflict and picks the newest version of the dependency, which is 1.1.0.

This situation can be illustrated as follows:

Remember the output we saw earlier?

com.msignoretto:lib:1.0.0 -> 1.1.0 (*)

This means Gradle promoted com.msignoretto:lib to version 1.1.0.

To facilitate the thought process you can interpret com.msignoretto:lib:1.0.0 -> 1.1.0 (*) as “Use definitions from version 1.0.0 and implementations from version 1.1.0.”

Understanding this fundamental concept is pivotal in comprehending how binary compatibility operates.

If you want to learn more about how Gradle resolves conflicts, you can check out the official documentation here.

A closer look at binary compatibility

Now that we have a basic understanding of Gradle’s dependency resolution, let’s delve deeper into the concept of binary compatibility.

Our library can be divided into two distinct parts:

The public APIs: This is the part that is exposed to the client and serves as the contract for interacting with the library. Generally, any public class, interface and so on are part of the public contract.
The implementation: This is the section that remains hidden from the client and is used solely for implementing the public APIs, usually those classes and functions have private, internal visibility.

For those acquainted with C programming, consider the first part as the .h header file included in the client code, representing the contract. Conversely, the second part resembles the .c file linked to the client code at compile time. This analogy is drawn from the clear separation between contract and implementation in C. Unlike Kotlin and Java, where these aspects are defined in the same place.

As a C programmer I could create perfectly encapsulated objects, without resorting to awful hacks like “public” and “private” modifiers.

In that sense C was a better OOPL than Java, C++, or C#.

Uncle Bob Martin

The potential source of binary compatibility issues resides within the first part only, the public APIs.

Let’s try to depict this situation visually using the aforementioned example.

As mentioned earlier, Gradle prioritizes the newest version of the dependency. However, it’s important to note that the code in moduleB was compiled using version 1.0.0 of the library. Consequently, it still depends on the previous version’s public API, meanwhile, the library has been updated to version 1.1.0.

The situation can be illustrated as follows.

Maintain binary compatibility

Now that we comprehend what binary compatibility is and how it can be compromised, let’s explore how we can make changes while preserving binary compatibility.

First of all, let’s examine the decompiled bytecode of the two versions of the library.

Upon decompiling version 1.0.0 of the DishesRepository interface into Java, we encounter the following code:

public interface DishesRepository {
   @NotNull
   List getDishes();
}

Similarly, decompiling version 1.1.0 of the DishesRepository interface reveals the following code:

public interface DishesRepository {
   @NotNull
   List getDishes(@NotNull DishSpecification var1);

   public static final class DefaultImpls {
      // $FF: synthetic method
      public static List getDishes$default(DishesRepository var0, DishSpecification var1, int var2, Object var3) {
         if (var3 != null) {
            throw new UnsupportedOperationException("Super calls with default arguments not supported in this target, function: getDishes");
         } else {
            if ((var2 &amp; 1) != 0) {
               var1 = DishSpecifications.INSTANCE.all();
            }

            return var0.getDishes(var1);
         }
      }
   }
}

The decompiled Java bytecode clearly illustrates that the second version of the library not only introduces a peculiar static class for default implementation but also incorporates a new method getDishes(@NotNull DishSpecification var1) that requires a DishSpecification as a parameter. Meanwhile, the previous version of the getDishes() method without any parameters, has been removed.

To address this specific case, a simple solution is to explicitly add a function without extra parameters and remove the default value, as demonstrated in version 1.2.0 of the DishesRepository:

public interface DishesRepository {
    public fun getDishes(): List<Dish>
    public fun getDishes(specification: DishSpecification): List<Dish>
}

Let’s observe the decompiled bytecode after implementing this change:

public interface DishesRepository {
   @NotNull
   List getDishes();

   @NotNull
   List getDishes(@NotNull DishSpecification var1);
}

You’ll notice that the old function getDishes() present in version 1.0.0 remains intact, while the new one is added. This modification maintains binary compatibility because the old code compiled against version 1.0.0 will still find the function getDishes() in the new bytecode.

Backward compatibility

Now that we’ve addressed binary compatibility, it’s essential to understand how to handle the old function. Although you might assume the work is complete, there’s another critical aspect to consider: backward compatibility.

Your clients expect the code to behave as it did before. Therefore, even if you’ve maintained binary compatibility, changing the behavior of the code in the new version could break backward compatibility.

Let’s illustrate this with the same code example used earlier.

In version 1.2.0, we reintroduced the function:

public fun getDishes(): List<Dish>

Let’s hypothesize you chose to simply return an empty list, considering this function obsolete and not intended for use anymore.

Upon running the code from the main file mentioned earlier, the output is now:

>> Module A <<
>> Module B <<

Contrast this with the previous output:

>> Module A <<
Pizza - Pizza with tomato and cheese - 10.0
Pasta - Pasta with tomato and cheese - 8.0
>> Module B <<
Pizza - Pizza with tomato and cheese - 10.0
Pasta - Pasta with tomato and cheese - 8.0

Backward compatibility refers to the preservation of the behavior of the old version even in the new version.

In this case, the behavior of the new code differs from the old one, breaking backward compatibility. It’s important to note that this change doesn’t just affect the code using the new version; it has a cascading impact, even on the module depending on the older version.

Tricks and tips

In this section, I’ll share some general recommendations and guidelines that I adhere to, and which can assist you in maintaining compatibility.

Use explicitApi(): By utilizing this feature, you’ll be compelled to explicitly define the visibility of your classes and interfaces. This approach fosters a deeper awareness of what constitutes the public API versus internal components. Remember, a concise public contract minimizes the likelihood of breaking binary compatibility, to learn more check the official documentation.
Utilize Compatibility Checking Tools: Incorporate tools like Metalava from Google to assess your code’s binary compatibility. Integrating the convenient Metalava Gradle plugin into your CI checks can streamline your workflow and shield you from unintended binary incompatible changes.
Segregate Implementation from API Modules: Keep implementation details separate from API modules to prevent the exposure of internal implementation details to clients and the leakage of such into the public API.
Exercise Caution with Kotlin Data Classes: While data classes are beneficial, they aren’t inherently binary compatible. When you want to expose a data class to clients, consider exposing a normal class with hashCode, equals, copy methods instead. You can find more insights on this in the Public API challenges in Kotlin article.
Handle Default Parameters Mindfully: While default parameters offer convenience, they aren’t binary compatible either. If you need to expose a function with a default parameter, consider creating two distinct functions as in the old Java days or explore the use of @JvmOverloads to manage defaults effectively.

Exercise Caution with Default Interface Methods: Default interface methods pose another challenge as they aren’t binary compatible as well. Delve deeper into this concept by reading the Kotlin Default Interface Methods and Binary Compatibility article.

Conclusion

We’ve reached the end of this article, and I sincerely hope you found it informative and enriching.

Binary compatibility is undoubtedly a multifaceted topic that might initially seem challenging to grasp. It’s important to recognize that even if you’re not directly involved in library maintenance, familiarizing yourself with this concept remains essential.

You might encounter instances where third-party dependencies that are used in your code lack binary compatibility, necessitating strategic handling on your part.

Consider scenarios where your company’s libraries are spread across different repositories, requiring a thorough understanding of binary compatibility to ensure smooth integration.

Similarly, if you decide to contribute to an open-source library, ensuring the compatibility of your changes becomes a critical aspect of your contribution.

Moreover, with the growing trend of micro-frontends in mobile app development, where various teams work on different components of the same application, maintaining binary compatibility becomes paramount for seamless integration and potential feature rollbacks without affecting the entire application.

Regardless of your specific role, binary compatibility is likely to affect you at some point in your career. I trust that this article has equipped you with a deeper understanding of the topic, enabling you to navigate and master binary compatibility when the need arises.