Building a flatpak app without the flatpak cli

12 minutes read. Published:

Table of contents

Introduction

I believe building a flatpak package is the easiest way to distribute an app targeting 90% of linux distros. Preparing a flatpak package mostly means writing a manifest.json describing all the needed dependencies of the app and calling flatpak-builder with the manifest file. Today we are going deeper. We won't write a manifest file, we won't use flatpak-builder and not even flatpak. We are going to replicate the build process manually, in the shell, to see how the technologies behind flatpak are cooperating.

This article was written to improve my knowledge on flatpaks and to evaluate potential changes/improvements to the flatpak build process

Starting from scratch

We are going to start from a clean system, without flatpak installed. To do so, I'm going to work inside a fedora virtual machine.

vagrant init fedora/37-cloud-base \
  --box-version 37.20221105.0
vagrant up
vagrant ssh

# I've also edited the generated Vagrantfile to grant more memory to the vm...

Getting access to flathub

We will need to download some data from flathub, that is, the biggest flatpak repository. Normally, if we had flatpak installed, we would follow the guide on https://flatpak.org/setup/, but we can't do that.

Let's try following the steps manually. We are interested in replicating the command

flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

For now we will just download the .flatpakrepo file using curl:

curl -L "https://flathub.org/repo/flathub.flatpakrepo" > flatpakrepo

Understanding repositories

Flatpaks must work on every distro. That means the app we are going to distribute needs to bring with it all the needed dependencies.

Let's say we want to build a python program. Python may not be installed on the target system, so we need to bring python with us. The same goes for all the dependencies of our app.

If we keep this line of reasoning, installing 10 flatpaks will require downloading a copy of python 10 times. That's completely wasteful and unnecessary. To eliminate this problem, flatpaks underneath are built using ostree repositories.

OSTree

OSTree offers a way to build a filesystem, track its changes overtime, and distribute it. The changes are stored in a "repository", similarly to how git works.

When we download 10 flatpaks which use the same python version, we aren't downloading python 10 times. We're downloading an ostree repository containing python, one time. When we run a program using flatpak run org.myorg.Myapp, the flatpak command takes the job of creating a "container" with the files of the ostree repository containing python and the files of our program.

In reality, ostree repositories often are a big bundle of dependencies, and they don't contain only python. Most apps are probably going to reuse the org.gnome.Runtime branch, which contains python, but also gjs (a javascript runtime) and many other commands and libraries needed to run a GTK app. There are also branches containing build tools, like org.gnome.Sdk, which are only used during build-time and aren't required by the end user installing our app.

Downloading the required branches

We are going to download the branches org.gnome.Runtime and org.gnome.Sdk from the repository flathub.

First, we need to init a local repository to store the contents of org.gnome.Runtime and org.gnome.Sdk. Oh, and let's install ostree.

dnf install ostree
ostree init --repo myrepository --mode bare-user-only

The flag bare-user-only is required so that the downloaded files can be used by an unprivileged user. The same mode is used by flatpak on the user folder located at ~/.local/share/flatpak. If you have already used flatpak on your system, you can in fact check that the file ~/.local/share/flatpak/repo/config contains 

[core]
...
mode=bare-user-only
...

To retrieve the download-url of the branches, we need to see the contents of the flatpakrepo file we have downloaded earlier:

[Flatpak Repo]
Title=Flathub
Url=https://dl.flathub.org/repo/
Homepage=https://flathub.org/
Comment=Central repository of Flatpak applications
Description=Central repository of Flatpak applications
Icon=https://dl.flathub.org/repo/logo.svg
GPGKey=mQINBFlD2sABEADsiUZUOYBg1UdDaWkEdJYkTSZD68214m8Q1fbrP5AptaUfCl8KYKFMNoAJRBXn9FbE6q6VBzghHXj...

We can see it contains a user friendly description of the repository, a url which we will use to download our data, and even an Icon which can be displayed by software stores, like KDE's Discover or the gnome software center.

Now we can add the Url we just found in the file flatpakrepo as a remote of our local repository, in a similar way to what we would do with git.

ostree remote add --repo myrepository flathub https://dl.flathub.org/repo/

We can't use the remote yet, because ostree wants to check the signature of everything we download from it. We need to import the GPGKey of the repository. The key can be found inside the file flatpakrepo , but it's in base64, so we need to decode it. To ease the process, let's copy-paste the GPGKey field to a file base64key. Then:

base64 --wrap 0 -d base64key > repokey
ostree remote gpg-import --repo myrepository flathub --stdin < repokey

We can now download the branches we need from flathub

ostree pull --repo myrepository --depth 1 flathub runtime/org.gnome.Sdk/x86_64/43
ostree pull --repo myrepository --depth 1 flathub runtime/org.gnome.Platform/x86_64/43

Working with the downloaded branches

If everything went well, we should be able to list the files inside one of the branches we just downloaded

ostree ls --repo myrepository runtime/org.gnome.Sdk/x86_64/43

The files are now stored inside the repository in a content-addressed-object store. To access those files "normally", we need to checkout the branches to a folder.

# create folders to store the checkouts
mkdir -p ./runtime/org.gnome.Sdk/x86_64/
mkdir -p ./runtime/org.gnome.Platform/x86_64/

# checkout the branches to their respective folders
ostree checkout --repo myrepository runtime/org.gnome.Sdk/x86_64/43 ./runtime/org.gnome.Sdk/x86_64/43
ostree checkout --repo myrepository runtime/org.gnome.Platform/x86_64/43 ./runtime/org.gnome.Platform/x86_64/43

# check that everything went ok
ls -la ./runtime/org.gnome.Platform/x86_64/43/

This process is similar to flatpak install: when you install a branch as an end user, using flatpak install org.gnome.Sdk, flatpak will checkout the files of that branch to a folder as /var/lib/flatpak/runtime/org.gnome.Sdk/x86_64/43/.

Creating the build environment

We now need to replicate flatpak build-init. This command creates a build folder containing some files, as follows:

mkdir -p build/files
mkdir -p build/var
touch build/metadata

The folder build/files will contain the files of our app. metadata is a description of the flatpak we are building, containing the icon path of our app, the required permissions, and other metadata...

We are mostly interested in filling build/files with something executable.

Entering the build environment

We are now replicating the command flatpak build. We will build a small GTK4 app, in C, so that we are sure the build environment is working correctly. I haven't installed any gtk library in the virtual machine, so if the build succeeds, it means we are using the libraries and the compiler provided by org.gnome.Sdk.

flatpak build ./build command would do the following:

We can do the same thing using bubblewrap, a tool to create a sandbox, or in the words, a container, on the fly. bubblewrap is heavily used by flatpak to achieve sandboxing.

Let's enter the build environment

dnf install bubblewrap
bwrap \
--unshare-all \
--clearenv \
--dev /dev \
--proc /proc \
--bind /home /home \
--bind ./runtime/org.gnome.Sdk/x86_64/43/files /usr \
--symlink usr/lib /lib \
--symlink usr/lib64 /lib64 \
--symlink usr/bin /bin \
--symlink usr/sbin /sbin \
--bind ./build/files/ /app \
--bind ./build/var/ /var \
/usr/bin/bash

Using --unshare-all we are telling bubblewrap to run the process bash with a different ipc, net, pid and user namespace. That means the process won't be able to use the network, it won't be able to see the other processes (except its children) and it will be assigned to a different uid.

You can see we are binding the sdk folder ./runtime/org.gnome.Sdk/x86_64/43/files to /usr. Then we have created some symlinks to /usr, to ensure that the binaries and the loader is accessible. The loader path is expected to be on /lib64/....

/app will contain our executable file.

We are now inside the container, running bash. We can check that we have access to every file by wandering around:

ls -la /
file /bin/bash

Before continuing, we should export the PATH env variable, else a lot of things will behave extremely weirdly. Apparently bash has a default PATH variable, but it's not exported.

export PATH
# check that everything works...
which gcc

We have access to gcc, so we can finally build our app.

Building the app

I'm going to take the GTK4 example app from gtk.org and copy it to a file inside /app/main.c. I'm adding a printf line, to see some output once I'll run the program from my virtual machine, which doesn't have a graphics stack.

cat > /app/main.c <<EOF
// Include gtk
#include <gtk/gtk.h>

static void on_activate (GtkApplication *app) {
  // Create a new window
  GtkWidget *window = gtk_application_window_new (app);
  // Create a new button
  GtkWidget *button = gtk_button_new_with_label ("Hello, World!");
  // When the button is clicked, close the window passed as an argument
  g_signal_connect_swapped (button, "clicked", G_CALLBACK (gtk_window_close), window);
  gtk_window_set_child (GTK_WINDOW (window), button);
  gtk_window_present (GTK_WINDOW (window));
}

int main (int argc, char *argv[]) {
  printf("HELLO WORLD!");
  // Create a new application
  GtkApplication *app = gtk_application_new ("com.example.GtkApplication",
                                             G_APPLICATION_FLAGS_NONE);
  g_signal_connect (app, "activate", G_CALLBACK (on_activate), NULL);
  return g_application_run (G_APPLICATION (app), argc, argv);
}

EOF

Now we need to build it. I'm now copying the build command from gtk.org, adapting it to our use case:

mkdir /app/bin
gcc $(pkg-config --cflags gtk4) -o /app/bin/main /app/main.c $(pkg-config --libs gtk4)
# check that everything went well
ls -la /app/bin/main
# remove the source file, so that we don't waste bytes in the final export
rm /app/main.c

We are done compiling. Let's exit the build environment with exit.

Finishing the build

At this point we would call flatpak build-finish ./build --command main. Let's replicate the command manually.

We would need to export the icon of our app from ./build/files to ./build/export, but we don't have an icon, so we'll skip that.

The most important thing to do is to fill the metadata file with something useful. We need to tell flatpak what's the entry point of our app. We also need to give it a name: my.handcrafted.App.

cat > ./build/metadata <<EOF
[Application]
name=my.handcrafted.App
runtime=org.gnome.Platform/x86_64/43
sdk=org.gnome.Sdk/x86_64/43
command=main
EOF

Exporting the app to a repository

flatpak installs apps from a repository, like flathub. Our app isn't inside a repository, so we can't install it yet.

To export the app to a local repo, we would use flatpak build-export. Let's create another ostree repo, manually. Then we commit our changes to a branch with the syntax app/APPNAME/ARCH/VERSION, so that it's discoverable by flatpak install.

ostree init --repo outrepo --mode bare-user-only
ostree commit --repo outrepo \
  --branch app/my.handcrafted.App/x86_64/master \
  --add-metadata-string xa.metadata="$(cat ./build/metadata)"$'\n' \
  --tree=dir=./build 
ostree summary -u --repo outrepo

Apparently the metadata file must be added also inside the metadata of the commit. 🤷 Else, we can't add the repository using flatpak remote-add. Some info on this issue.

Notice also the trick "..."$'\n' in the parameter --add-metadata-string. I'm using that to add a newline, which is present inside ./build/metadata, but seems to be discarded by the process substitution "$(cat ./build/metadata)". 🤷

Installing the app and running it

This is the last step. The app is already built and ready to be used. We can finally use flatpak. Yay!

It's the same process as if we were installing the app from flathub.

dnf install flatpak
# install the usual flathub repo, that's needed to download org.gnome.Sdk and org.gnome.Platform, as with any other app
flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install --user org.gnome.Sdk 
flatpak install --user org.gnome.Platform

# Add our local repository containing the app
flatpak remote-add --user --no-gpg-verify outrepo file://`pwd`/outrepo

# Finally install and run it
flatpak install --user outrepo my.handcrafted.App
flatpak run --user my.handcrafted.App

And then we get...

[vagrant@localhost ~]$ flatpak run --user my.handcrafted.App
Failed to register: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: org.freedesktop.DBus.Error.ServiceUnknown
HELLO WORLD![vagrant@localhost ~]$

OH YEAH!

(we don't care about that dbus error, it's nothing important, I guess it's not even related to our build process).

WE'VE DONE IT

Conclusions

That was a long ride. We've seen how ostree is an essential component of flatpak. It enables distribution of apps, runtimes and sdks using repositories, and it enables keeping track of changes in a git-like fashion. The flatpak command also relies heavily on namespaces provided by the kernel, in a similar way to docker and podman. In a certain sense, we could say docker/podman is used for running contanerized services on servers, and flatpak is used for running containerized apps on personal computers. Bundling the most common dependencies inside a runtime, as org.gnome.Platform/x86_64/43, is certainly a smart move, because it enables security updates by updating only the runtime, without needing to rebuild every single app. ostree pull is also smart enough to download only the changed files, so doing a security update won't require a giant update.

The build process isn't easy, nor small, but I can see that the complexity is mostly inevitable. The tool flatpak-builder tries to hide most of the complexity by making the build process declarative, but it expects users to declare everything inside a static manifest.json file, which could be limiting for some people. Also, it requires learning the syntax and rules of the file manifest.json, which is quite an effort itself. But flatpak-builder isn't the scope of this article, so I'm going to stop here.

Thanks for reading. If you liked this post, you can buy me a coffee on patreon or github.