Learn OSGi from Scratch — Eclipse, IntelliJ and WSO2 Platform 🖥️

Consider the following question. In any large engineering project, for example, the design of a new bridge, or jet airliner, what is the most difficult challenge to overcome?

The answer is 💥 Complexity.

The Lamborghini Aventador has around one million parts and it is a very, very complex machine, and because of this complexity, no single person can have a complete understanding of how it works. Despite that it was build and its complexity is getting multiplied every year. So how can engineers build such a machine?

The answer is breaking the machine into smaller, more understandable modules.

Modularity enables several important benefits.

• Division of Labour — Can assign separate individuals or groups to work on separate modules. The people working on a module will have a thorough understanding of their own module.
• Abstraction — Consider the Aventador as an abstract model. We can get the idea that it will move without having to grasp the idea of how fuel is supplied or how the engine works.
• Reuse — Given the amount of time it takes to design even smaller components of the Aventador, it would be a waste to start them from scratch when we need a similar component in another car. Therefore, it would be helpful if we could reuse the components with minimal alterations.
• Ease of Maintainance and Repair — It would be crazy to rebuild the whole car whenever it gets a flat tire. Modular designs allow for failed modules to be removed and either repaired or replaced without affecting the rest of the machines.

The Java programming language is one of the most popular languages for building large, enterprise applications and also small but widely deployed mobile applications. However, Java alone does not support modularity in any useful way. But the Java’s flexibility and scalability have allowed a powerful module system to be built on top.

That module system is called OSGi (Previously the acronym for Open Service Gateway Initiative), pronounced as (Oh-ess-gee-eye).

🚀 What is a Module?

So, what is a software module? A software module is something that has the following properties.

• Self-Contained — A module is logical overall. It can be moved around, installed, and uninstalled as a single unit. It consists of smaller parts and those parts are integral to a module. If one of them is removed, the module may cease to function.
• Highly Cohesive — Cohesion is a measure of how strongly related the responsibilities of a module are. A module should not do many unrelated things, but stick to one logical purpose and fulfill it. ( e.g. — A networking module should not check the persistences, database verifications, input verifications. It should stick to the networking part only.)
• Loosely Coupled — A module should not be concerned with the internal implementations of other modules that it interacts with. Loose coupling allows us to change the implementation of one module without needing to update all other modules that use it.

To support all three properties, it is essential for modules to have a well-defined interface for interaction with other modules. A stable interface enforces logical boundaries between modules and prevents access to internal implementation details.

🚀 The Problems with JARs

The standard unit of deployment in Java is the JAR file. JARs are archive files based on the ZIP file format, allowing many files to be aggregated into a single file.

Usually, the files contained in the archive are a combination of compiled Java classes and resource files such as images. In addition, there is a standard location within the JAR archive for metadata — the META-INF folder which can contain many files with different formats, while MANIFEST.MF file is the most important file.

JAR files provide either a single library or a portion of the functionality of an application. Therefore, constructing Java applications requires combining many JAR files.

Yet, the Java Development Kit(JDK) provides only very rudimentary tools for managing and composing JARs. In fact, those tools are so simplistic that the term “JAR Hell” has been made by developers for the problem of managing JARs.

The most critical problems with JAR files as a unit of deployment are as follows:

• There is no runtime concept corresponding to a JAR; they are only meaningful during build-time and deploy-time. Once the JVM is running the contents of all the JARs are simply taken as a single, global list: Classpath.
• They don’t have standard metadata to indicate dependencies.
• They are not versioned and hence multiple JARs cannot be loaded simultaneously.
• There is no mechanism for information hiding between JARs.

ClassLoading and the Global Classpath

The term classpath comes from the command-line parameter that can be passed to the java command when running simple Java applications from the command shell. It specifies a list of JAR files and directories containing compiled Java class files.

For example, the following command launches a Java application with both log4j.jar and the classes directory on the classpath. The UNIX/macOS X command is:

java -classpath log4j.jar:classes org.example.HelloWorld

The final parameter is the name of the main class to execute, which is compiled to the org/example/HelloWorld.class in the classes directory per our assumption.

The JVM’s responsibility is to load the bytes in that class file and transform them into a Class object, then it can execute the static main method. Let’s look at how this works in a standard JRE.

The class in Java that loads classes is java.lang.ClassLoader and it has two responsibilities.

• Finding classes, i.e. the physical bytes on disk, given their logical class names.
• Transforming those physical bytes into a Class object in memory.

When we run the command, java -classpath log4j.jar:classes org.example.HelloWorld the JRE determines that it needs to load the class org.example.HelloWorld Since it is the main class, it uses a special ClassLoader named the application class loader. The first thing the application class loader does is ask its parent to load the class.

This process is a critical feature in Java class loading called parent-first delegation. To illustrate the whole process in a simple manner we will use the flow chart given below.

fig. 1 — The JRE class loading

In the chart there are three class loaders, the bootstrap class loader is at the top of the tree, and it is responsible for loading all the classes in the base JRE library ( java , javafx , etc.)

Next the extension class loader loads from extension libraries, the libraries which are not a part of the base JRE but installed to libext directory of the JRE by an administrator. Finally, there is the application class loader which loads the classpath.

1. The JRE asks the application class loader to load a class
2. The application class loader asks the extension class loader to load the class.
3. The extension class loader asks the bootstrap class loader to load the class.
4. The bootstrap class loader fails to find the class, so the extension class loader tries to find it.
5. The extension class loader fails to find the class, to the application class loader tries to find it, looking first in log4j.jar
6. The class is not in log4j.jar therefore the class loader looks in the classes directory.
7. If the class is found and loaded it will load the other classes starting from step 1 again. Else, it will give the common ClassNotFoundException.

Conflicting Classes

The loading classes in Java works most of the time without a hitch, but what would happen if there we add an obsolete JAR containing an older version of HelloWorld . Let’s call that JAR file obsolete.jar

java -classpath obsolete.jar:log4j.jar:classes org.example.HelloWorld

Since, obsolete.jar appears before classes in the classpath, and since the application class loader stops asap when it finds a match, this command will always have the same impact as using the old version of HelloWorld and the classes the directory will never be used.

Lack of Explicit Dependencies

Although there are some standalone JAR files that do not depend on other JAR files, most of the JAR files depend on other JAR files. If there is no proper documentation of how to use those JAR files, the functionality is going to be detrimental.

Lack of Version Information

The world does not stand still, and neither do the libraries. They get updated and getting new versions all the time. Therefore, indicating a dependency or a library is not enough. We need to know the exact version that we need. Most of the time the documentations help us not to fall into those version pitfalls. But sometimes we get issues like the example mentioned below:

Assume there are three JAR files, A.jar, B.jar, and C.jar. The A.jar wants the 1.1 version of C.jar and the B.jar wants the 1.2 version of the C.jar. And our application wants A.jar and B.jar.

In this case, as the classpath only selects one version of C.jar we get a dependency issue with either A.jar or B.jar. Therefore it is evident that these kinds of problems cannot be solved with traditional Java without rewriting the source codes related to either A.jar or B.jar.

Lack of Information Hiding Across JARs

All Object Oriented Programming languages offer different ways of hiding information. In Java, the encapsulation depends on the access modifiers we use.

• public — members are visible to everybody.
• protected — members are visible to subclasses and other classes in the same package.
• private — members are visible only within the same class
• default — members which are not declared with the above three access levels. They are only visible to other classes within the same package, but not outside the package.

As a result, all those classes declared public is accessible to clients outside the JAR. Therefore the whole JAR is basically public API, even the parts that we would prefer to keep hidden.

Therefore, JARs are not modules. Although they can be moved around, they have tightly coupled ZIP archives with low cohesion.

But this does not say that we can’t build a modular system without using JARs. JARs are needed for the implementation of a modular system but they are not the modules.

🚀 J2EE Class Loading

The Java 2 Enterprise Edition(J2EE) specification defines a platform for distributed, multi-tier computing. The critical feature of the J2EE architecture is the application server, which hosts multiple application components and offers enterprise-level service to them.

These requirements insinuate that the simplistic class loading diagram in fig.-1 is not sufficient. Because with a single flat classpath the classes from one application could easily interfere with other applications. Therefore J2EE uses a more advanced class loading hierarchy which is basically a tree with a branch for each deployed application.

J2EE applications are deployed as Enterprise ARchive (EAR) files which are ZIP files containing a metadata file — application.xml plus one or more of the following files:

• Plain Java Library JAR files
• JAR files containing an (Enterprise Java Beans)EJB application (EJB-JARs)
• Web ARchive (WAR) files, containing classes implementing Web functionality such as servlets and JSPs.

But there are issues in this architecture too, the class sharing between the upper-class loaders(not the ones in branches)can lead to conflicts with versioning.

fig. 2 — J2EE class loading

🚀 OSGi

OSGi is the module system for Java. It defines a way to create true modules and a way for those modules to interact at runtime.

The main idea of OSGi is quite simple. The source of most of the problems in traditional Java is the global, flat classpath. So, OSGi takes a different approach: each module has its own classpath separate from the classpath of all other modules.

This eliminates almost all the problems we discussed earlier. But this doesn’t end here, we still need our modules to work together which means sharing the classes. OSGi has very specific and well-defined rules on how to share classes across modules, using a mechanism of explicit imports and exports.

So, what does an OSGi module looks like? First, we don’t call it a module, we call it a bundle in OSGi. Indeed it is just a JAR file! But it contains the metadata to promote it to a bundle. The metadata consists of:

• The name of the bundle.
• The version of the bundle.
• The list of imports and exports.
• Optionally, the info on the minimum Java version that the bundle needs to run on.
• Miscellaneous human-readable info such as vendor, copyright statement, contact address, etc.

These metadata are placed inside the JAR file in a special file called MANIFEST.MF, which is a part of all standard JAR files and is meant for exactly this purpose. And since bundles are JAR files they can be used outside the OSGi runtime.

Trees vs. Graphs

In OSGi, it provides a separate classpath for each bundle. It simply means that we provide a class loader for each bundle and that the class loader can see the classes and resources inside the bundle’s JAR file. However, in order for bundles to work together, there should be a way to load classes delegated from one bundle’s class loader to another.

In both fig. 1 and fig. 2 class loaders are arranged in a hierarchical tree, and the class loading requests are always delegated upwards, to the parent of each class loader and those two trees did not have a way to do horizontal delegations. To make a library available to multiple branches it must be pushed up into the common ancestor of those branches.

But in OSGi, it is solved by using a graph. The dependency relationship between two modules is not a hierarchical one: there is no parent, child delegations, only a network of providers and users. Classloading requests are delegated from one bundle to another based on the dependency relationship between the bundles.

fig. 3 — OSGi class loader graph

The links between bundles are based on the imported and exported packages.

For example, suppose that bundle B in fig. 3 contains a package named org.foo It may choose to export that package by declaring it in the exports section of its MANIFEST.MF. Bundle A then chooses to import org.foo by declaring it in the imports section of its MANIFEST.MF. Now, the OSGi framework will match the import with a matching export: this is known as the resolution process. Once an import is matched up with export, the bundles involved are wired together for that specific package name. What this means is that when a class load request occurs in bundle A for any class in org.foo package, that request will immediately be delegated to the class loader of bundle B.

So, what happens if resolution fails? In that case, bundle A will not resolve and cannot be used. Assuming that our two bundles A and B are correctly constructed, we would not see any errors like ClassNotFoundException or NoClassDefFoundError in OSGi-based applications. In fact, it will tell the start-up that something is wrong. Therefore, when using OSGi we can know about resolution errors in a set of bundles before we never execute the application.

Versioning and Side-by-Side Versions

OSGi does not merely offer dependencies based on package names, it also gives versioning of packages. This allows coping with changes in the released versions of libraries we use.

Exports of packages are declared with a version attribute, but imports declare a version range. This allows us to have a bundle depend on e.g. version 1.1.0 through to version 2.1.0 of a library. If no bundle is exporting a version of that package within that range then the bundle will fail to resolve and we will get a helpful error message telling us what is wrong.

We can even have different versions of the same library side-by-side in the same application.

🚀 OSGi Architecture

The OSGi platform is composed of two parts: OSGi Framework and OSGi Standard Services.

OSGi Framework

The OSGi framework plays a critical role when you create OSGi-based applications. And there are three conceptual layers defined in the OSGi specification.

• Module layer — concerned with packaging and sharing code
• Lifecycle layer — concerned with providing execution-time module management and access to underlying OSGi framework
• Service layer — concerned with interaction and communication among modules, specifically the components contained in them.

OSGi Standard Services

The standard services define reusable APIs for common tasks, such as logging.

OSGi Implementations

Several independently implemented OSGi frameworks exist today, including four that are available as open-source software.

• Equinox
• Knopflerfish
• Felix
• Concierge

🚀 First Steps in OSGi

OSGi Development Tools

In theory when building OSGi bundles one does not need any additional tools except the standard Java tools: javac for Java source code compilation, jar for packaging, and a text editor for creating the MANIFEST.MF file.

However, it is laborious to use these basic tools since they require lots of effort. Therefore, in practice, we use build tools like Ant or Maven, and IDEs like Eclipse, NetBeans, or IntelliJ.

For this phase, we will use Eclipse IDE. But in a later phase, we will show how to use IntelliJ to build an OSGi project.

Getting a Framework

As we mentioned before there are four open-source OSGi implementations — Equinox, Knopflerfish, Felix, and Concierge. We will mainly work with Equinox for this tutorial. The download link is given below:

Equinox Downloads | The Eclipse Foundation

Download the latest SDK from the download page and extract it to a folder like equinox-SDK. We will refer this top-level directory as EQUINOX_HOME . After decompressing we will have a directory named plugins where we can find all the JARs that implement Equinox and its supporting bundles. In Eclipse, you have the Equinox framework in the underlying runtime environment.

Project in Eclipse 🌓

Open Eclipse IDE and go to New → Project → Plug-in Project

fig.4 — setting up eclipse project

Give the project name OSGi Tutorial and select Equinox as the OSGi framework.

fig. 5 — setting up eclipse project

Then click Next > and select the execution environment. For this select the Java version you installed into your computer. And select the checkbox for generate an activator. And then click Finish

fig. 6 — setting up eclipse project

fig.7 — project overview

Hello, World!

In keeping with the long-standing tradition, our first program in OSGi will be one that simply prints “Hello World!” to the console. However, most such programs exit as soon as they print the message. But we will extend the tradition with not only printing “Hello” upon start-up but also “Goodbye” upon shutdown.

In our project, go to src → osgi_tutorial → Activator.java file and rename it to HelloWorldActivator.java And then replace the code with the following code snippet.

Since we changed the Activator.java to HelloWorldActivator.java we should change it in the MANIFEST.MF file too. Go to MANIFEST.MF file and change the osgi_tutorial.Activator to osgi_tutorial.HelloWorldActivator

fig. 8 — changing MANIFEST.MF file

It is notable that you can view the MANIFEST.MF file using the bottom tab-pane too. Since making the MANIFEST.MF is the most important part of OSGi projects better to know all the options related to MANIFEST.MF file editing.

fig. 9 — MANIFEST.MF file

Now click the OSGi Tutorial from the left pane and right-click and Run As → OSGi Framework

fig. 10 — running OSGi project

When you do that you will get a bunch of error messages in the console like this.

fig. 11 — console output

After that check whether you are using the osgi console. If you are using the osgi console you will see something like osgi > in the console. Type ss to see all the bundles that are running in the project. And you will see something like this with ids.

fig. 12 — ss to view all bundles

And you can see that out OSGi_Tutotial_1.0.0.qualifier is there with the id 2 Now you can simply stop this by typingstop 2 and start it again by typing start 2 in the osgi console.

fig. 13 — starting and stopping the project

As you noted this start and stop are related to the methods we have written in HelloWorldActivator.java file.

Bundle Lifecycle

It was mentioned that OSGi bundles have a lifecycle, but what exactly is that lifecycle?

Our OSGi_Tutorial bundle starts with the install command and it enters to INSTALLED state. Then with the start command, it transitions to the ACTIVE state. Although we can’t see a direct link between the two states, bundles can only be started when they are in RESOLVED state. However, when we attempt to start an INSTALLED bundle, the framework tries to resolve it first before proceeding to start it.RESOLVED means that the constraints of that bundle are been satisfied. After being resolved it passes through STARTING to ACTIVATE state. The STARTING state is a transient state.

When the stop command is executed, the bundle transitions to RESOLVED state while passing the transient state STOPPING

To get more info on the lifecycle, check the diagram given below.

fig. 14 — bundle lifecycle

The BundleContext in our HelloWorldActivator.java file allows us to do multiple things like,

• Look up system-wide configurations.
• Find another installed bundle by its ID.
• Obtain a list of all installed bundles.
• Install new bundles programmatically.
• Register and unregister bundle listeners.
• Register and unregister service listeners.

Therefore simply changing the Activator class we can do several things.

🚀 Bundle Dependencies

As we pointed out earlier, managing dependencies is the key to achieve modularity. This can be a big problem in Java and many other languages as well since only a handful provide the kind of module systems that are needed to build large applications. The default module system in JAVA is the JAR-centric “classpath” model, which fails mainly not due to its inability to manage dependencies but instead leaving them up to chance.

OSGi takes away the element of chance by managing dependencies so that they are explicit, declarative, and versioned.

• Explicit — A bundle’s dependencies are in the open for anybody to see rather than hidden in a code path inside a class file, waiting to be found at run time.
• Declarative — Dependencies are specified in a simple, static, textual form for easy inspection. A tool can calculate which set of bundles are required to satisfy the dependencies of a particular bundle without actually installing or running any of them.
• Versioned — Libraries change over time, and it is not enough to merely depend on a library without regard to its version. OSGi, therefore, allows all inter-bundle dependencies to specify a version range, and even allows for multiple versions of the same bundle to be present and in use at the same time.

🚀 Brace your selves, OSGi is coming!

Now we got an understanding of OSGi, let’s try to create a Maven-based project using OSGi using IntelliJ. The reason we use IntelliJ here is that to show that OSGi is independent of the IDE.

We will create a project named, book-inventroyusing IntelliJ plus OSGi.

In this project, the reader(consumer) requests to read a book, and the book provider(producer) finds that book(generates) and lets the reader read it. Now to map the above scenario, we need different modules. Therefore, the reader module will have functionalities like requesting a book, viewing book details, etc. and the provider module will have functions like generating a book, updating a book, etc.

The most important task in building an OSGi project is to write the MANIFEST.MF file we talked about above. In most scenarios, writing it from scratch is not easy. Therefore we use the Maven Bundle Plugin for that. Now let’s get our hands dirty with some coding.

Go to New Project → Maven to create a new Maven project. And select Next

fig. 15 — new maven project

For the project, we will give the following details,

groupId: org.wso2.carbon
artifactId: book-inventory
version: 1.0-SNAPSHOT
packaging: pom
name: WSO2 Carbon-Book Inventory

fig. 16 — book-inventory project

Since the book-inventory is our parent module, we can delete the auto-generated src folder. We actually do this because we are not exporting the parent module. That is the reason we give packaging as pom

After creating the parent module, we should create two submodules, org.wso2.carbon.book.reader and org.wso2.carbon.book.provider To create these modules, right-click on the parent project and select Add New → Module After adding those two sub-modules, the project will look like this(Ignore the pom.xmlfiles.)

fig. 17 — project structure

When adding sub-modules to the project, the parent project pom.xml should be updated with <modules> and the sub-modules pom.xml will have <parent> tags. If you are using an IDE, it auto generates them.

The final pom.xml of the book-inventory will look like this:

Book Provider

The org.wso2.carbon.book.provider will have the following specifications

artifactId: org.wso2.carbon.book.provider
packaging: bundle
plugin: maven-bundle-plugin, maven-scr-plugin
dependency: org.apache.felix.scr.ds-annotations, org.eclipse.osgi.services, org.eclipse.osgi
name: WSO2 Carbon-Book Provider

Here, <packaging>bundle</packaging> is used since we need this to be bundled in and you can see that we are using maven-bundle-plugin

In maven-bundle-plugin we identify the following:

• Bundle-Symbolic-Name
• Bundle-Name
• Export-Package
• Import-Package
• Private-Package

Here, you can see that,

<private-package>org.wso2.carbon.book.provider.internal</private-package>
<Export-Package>
    !org.wso2.carbon.book.provider.internal,
    org.wso2.carbon.book.provider.*
</Export-Package>

is used, since we don’t want internal package to be exported.

Let’s start creating the files related to provider 🧑🏻‍💻

fig. 18 — project structure

Now, we will register the created service of the book provider in a registry to use it from other components. That is done via BookProviderServiceComponent.java This is similar to the Activator we discussed above when creating the OSGi project in Eclipse.

Book Reader

Theorg.wso2.carbon.book.reader will have the following specifications.

artifactId: org.wso2.carbon.book.reader
packaging: bundle
plugin: maven-bundle-plugin, maven-src-plugin
dependency: org.apache.felix.src.ds-annotations, org.eclipse.osgi.services, org.eclipse.osgi, org.wso2.carbon.book.provider
name: WSO2 Carbon-Book Reader

Here, you can see that, the provider is imported.

<Import-Package>
    org.osgi.framework; version="${osgi.framework.imp.pkg.version.range}",
    org.osgi.service.component; version="${osgi.service.component.imp.pkg.version.range}",
    org.wso2.carbon.book.provider.*; version="${project.version}"
</Import-Package>

We will have the following file structure in this sub-module.

fig. 19 — project structure

The code snippets of the files are given below.

After that, type maven clean install -DskipTests or simply maven clean install on the terminal to build the respective JARs.

fig. 20 — build success in terminal

Now, you will be able to see a newly generated folder in org.wso2.carbon.book.reader and org.wso2.carbon.book.provider named target Inside those folders there are JARs org.wso2.carbon.book.reader-1.0-SNAPSHOT.jar and org.wso2.carbon.book.provider-1.0-SNAPSHOT.jar If you unzip them you can see the files inside them. They contain the code and the MANIFEST.MF files and OSGI-INF files.

🚀 WSO2 Identity Server

Now we are going to run the OSGi project we created in the WSO2 platform. Go to https://wso2.com/identity-server/ and download the latest IS version. After extracting you will get a folder. We will call it <IS_HOME>

Go to <IS_HOME>/repository/components/dropins and paste the org.wso2.carbon.book.reader-1.0-SNAPSHOT.jar and org.wso2.carbon.book.provider-1.0-SNAPSHOT.jar inside it. Then go to the <IS_HOME>/bin via console and start the Identity Server.

In Linux/macOS → sh wso2server.sh -DosgiConsole

In Windows → wso2server.bat -DosgiConsole

fig. 21 — starting wso2-is

And you can see that our bundles are getting activated and working properly.

To stop wso2-is you can simply press CTRL+C .

🚀 OSGi Commands

There are few OSGi commands that we should be familiar with.

• ss → List down the bundles in the server with bundle id.
• ss <name> → Search the given name in the bundle and list it out.
• ls → List down services.
• b <id> → Show bundle info.
• diag <id> → Show unsatisfied constraints of the bundle.

🚀 References

These are the books I have read to create this article.

• OSGi in Practice — Bartlett N.
• OSGi and Equinox — Creating Highly Modular Java Systems — Jeff McAffer, Paul VanderLei, Simon Archer
• OSGi in Action — Richard S. Hall, Karl Pauls, Stuart McCulloch, David Savage

That’s all on how to work with OSGi on IntelliJ or Eclipse. The link to the IntelliJ project is given below.

GitHub - nipunaupeksha/osgi-book-inventory

Happy coding! 🧑🏻‍💻😇