Pine fun - Kernel skeleton

January 28 2021 by Norman Feske

Of the several kernels supported by the Genode OS framework, the so-called base-hw kernel is our go-to microkernel for ARM-based devices. Section Execution on bare hardware of the Genode Foundations book goes into detail about its underlying software design. This article describes the process of porting this kernel to a new board, specifically the Pine-A64-LTS single-board computer.

Equipped with the bare-metal serial-output facility developed in the previous article, we are eager to turn our attention to the kernel. Before attempting the porting of the kernel to the new board, however, it is recommended to run it first on one of the already supported boards to have a working reference. In the case of the Pine-A64 board, which is based on an Allwinner multi-core 64-bit ARM SoC, the closest approximation would be the NXP i.MX8Q EVK board, which ticks the boxes ARM, multi-core, and 64-bit. At the very least, one should give the kernel a try using Qemu's virtual pbxa9 board, which is a 32-bit platform. Even though this board has not much in common with ours, it is still useful for seeing how the various bits and pieces described below are supposed to work together.

Getting acquainted with the code base

The starting point of our line of work will be the existing board support for the i.MX8Q EVK. To get an idea of the amount of work ahead of us, let's examine the base-hw source tree within Genode for occurrences of the board's name. The search pattern "imx" is a good start.

 $ find repos/base-hw -type f | grep imx8
 repos/base-hw/lib/mk/spec/arm_v8/core-hw-imx8q_evk.mk
 repos/base-hw/lib/mk/spec/arm_v8/bootstrap-hw-imx8q_evk.mk
 repos/base-hw/recipes/src/base-hw-imx8q_evk/hash
 repos/base-hw/recipes/src/base-hw-imx8q_evk/content.mk
 repos/base-hw/recipes/src/base-hw-imx8q_evk/used_apis
 repos/base-hw/src/bootstrap/board/imx8q_evk/platform.cc
 repos/base-hw/src/bootstrap/board/imx8q_evk/board.h
 repos/base-hw/src/include/hw/spec/arm_64/imx8q_evk_board.h
 repos/base-hw/src/core/board/imx8q_evk/board.h

We can ignore everything inside the recipes/ directory for now. This directory contains package descriptions. We will come back to the packaging topic later. A grep -v hides these files from our view.

 $ find repos/base-hw -type f | grep imx8 | grep -v recipes
 repos/base-hw/lib/mk/spec/arm_v8/core-hw-imx8q_evk.mk
 repos/base-hw/lib/mk/spec/arm_v8/bootstrap-hw-imx8q_evk.mk
 repos/base-hw/src/bootstrap/board/imx8q_evk/platform.cc
 repos/base-hw/src/bootstrap/board/imx8q_evk/board.h
 repos/base-hw/src/include/hw/spec/arm_64/imx8q_evk_board.h
 repos/base-hw/src/core/board/imx8q_evk/board.h

On the one hand, it is nice to see such a small number of files to be concerned about. On the other hand, those files appear quite scattered throughout the source tree with a deep hierarchy, which is a bit confusing. To lift the clouds, let's have a look at the source-tree structure.

The files appearing under lib/mk/ are build-description files for libraries. There are two such files, having the file extension .mk. They are located in a sub directory called spec/arm_v8/, which means that the build system considers them only when building for an instruction set architecture that matches ARMv8.

Distinction between bootstrap and core

Given the set of files depicted above, we can immediately spot two construction sites, namely "bootstrap" and "core". The distinction between those two parts is illustrated in the following picture.

The bootstrap program is started by the boot loader while the CPU is running in physical mode. The MMU is disabled at this point. Only one CPU - usually referred to as the boot CPU - is active. Bootstrap is tasked with all the dirty and quirky work needed in preparation to bring up the so-called core component. This involves board-specific trickery like tweaking clocks and voltages, setting up the page tables for executing the core program in virtual memory, enabling the MMU, the initialization of additional CPU cores, and the ELF-loading of the core ELF executable. Once these steps are taken, bootstrap passes the control to the core component and ceases to exist.

The core component contains the microkernel executed in privileged mode. When using Genode on a traditional microkernel like NOVA or seL4, core is the first user-level program started by the kernel. It is usually called roottask. In contrast, when using base-hw as we are going to do now, core and the kernel are one single program. Core is the microkernel at the root of Genode's component tree. Hence, in the following, the terms core and kernel are used synonymously.

Core is executed with the MMU enabled. It is globally mapped at the upper part of the virtual address space. To operate as the kernel, it contains basic drivers for the interrupt controller, kernel timer (for preemptive scheduling), cache maintenance, and cross-CPU synchronization. For the interplay with the user level components running on top of core, it features code paths for exiting the kernel into the user land and, vice versa, for entering the kernel from the user land (syscalls, exceptions, interrupts). Functionality-wise, it implements mechanisms for inter-component communication, asynchronous notifications, physical-memory allocation, the management of virtual address spaces, and the world-switching between virtual machines (if used as a hypervisor). In short, everything a microkernel needs to do and - more importantly - nothing a microkernel shouldn't do.

Review of the board-specific code

Before starting the work on the new board support, let us briefly look into each of the files for the existing i.MX8q EVK board. Genode's support for the NXP i.MX family is hosted in the dedicated genode-imx repository. Let's draw our attention to the files named after board.

 repos/imx$ find | grep imx8q_evk

In the list of files, we spot three header files, one board.h header under src/bootstrap/, one board.h header under src/core/, and one imx8q_evk_board.h header under src/include/. The former two files are specific for bootstrap and core, whereas the latter contains definitions useful for both programs. The board.h files are located in directories named after the board. With this structure, generic (board-agnostic) code can #include <board.h>. The build system picks the right board.h file by adding the board-specific directory to the include-search path.

Let us start with with definitions used across bootstrap and core.

 #include <drivers/uart/imx.h>
 #include <hw/spec/arm/boot_info.h>

 namespace Hw::Imx8q_evk_board {
     using Serial = Genode::Imx_uart;

     enum {
         RAM_BASE   = 0x40000000,
         RAM_SIZE   = 0xc0000000,

         UART_BASE  = 0x30860000,
         UART_SIZE  = 0x1000,
         UART_CLOCK = 250000000,
     };

     namespace Cpu_mmio {
         enum {
             IRQ_CONTROLLER_DISTR_BASE  = 0x38800000,
             IRQ_CONTROLLER_DISTR_SIZE  = 0x10000,
             IRQ_CONTROLLER_VT_CPU_BASE = 0x31020000,
             IRQ_CONTROLLER_VT_CPU_SIZE = 0x2000,
             IRQ_CONTROLLER_REDIST_BASE = 0x38880000,
             IRQ_CONTROLLER_REDIST_SIZE = 0xc0000,
         };
     };
 }

Both bootstrap and core need to know the memory-mapped device registers for the UART device to print diagnostic messages. The UART driver (drivers/uart.imx.h) is included. The Serial type refers to the concrete UART driver implementation as present on the board. Thanks to this definition, generic code is able to rely on the UART functionality via the type name Serial.

The start and size of physical memory must be known by both bootstrap and core. So it is defined here.

Both bootstrap and core access the interrupt controller. Whereas bootstrap performs the one-time initializations needed in order to start secondary CPU cores, core drives the interrupt controller at runtime.

The bootstrap-specific files concern build descriptions and actual code. The build description looks as follows.

 REP_INC_DIR += src/bootstrap/board/imx8q_evk

 SRC_CC  += bootstrap/board/imx8q_evk/platform.cc
 SRC_CC  += bootstrap/spec/arm/gicv3.cc
 SRC_CC  += bootstrap/spec/arm_64/cortex_a53_mmu.cc
 SRC_CC  += lib/base/arm_64/kernel/interface.cc
 SRC_CC  += spec/64bit/memory_map.cc
 SRC_S   += bootstrap/spec/arm_64/crt0.s

 NR_OF_CPUS = 4

 vpath spec/64bit/memory_map.cc $(call select_from_repositories,src/lib/hw)

 vpath bootstrap/% $(REP_DIR)/src

 include $(call select_from_repositories,lib/mk/bootstrap-hw.inc)

The i.MX8 SoC uses the GICv3 as interrupt-controller. Hence, the driver gicv3.cc is included. In contrast, as we learned from the Linux boot log, the Allwinner A64 SoC uses the GICv2 interrupt controller.

The MMU driver differs between the various ARM versions. The i.MX8 is based on A53 CPU cores. The Allwinner A64 uses the same.

The assembly file arm_64/crt0.s contains the entry point into the program as jumped to by the boot loader.

The NR_OF_CPUS definition is used for the static allocation of data structures that must be present for each CPU. Hence, this value is globally defined.

The strange looking $(call select_from_repositories...) is a mechanism for accessing files across different source repositories. You can find the mechanism described in the manual.

 #include <hw/spec/arm_64/imx8q_evk_board.h>
 #include <hw/spec/arm_64/cpu.h>
 #include <hw/spec/arm/gicv3.h>
 #include <hw/spec/arm/lpae.h>

 namespace Board {
     using namespace Hw::Imx8q_evk_board;

     struct Cpu : Hw::Arm_64_cpu
     {
         static void wake_up_all_cpus(void*);
     };

     using Hw::Pic;
 }

The Board namespace aggregates the knowledge of the board details that matter to the bootstrap code, namely the specific interrupt controller (gicv3.h) and the declaration of the wake_up_all_cpus function. The Board namespace hosts the Pic (programmable interrupt controller) type, which allows the generic code of bootstrap to interact with the interrupt controller without knowing the exact type of device.

 Bootstrap::Platform::Board::Board()
 :
   early_ram_regions(Memory_region { ::Board::RAM_BASE, ::Board::RAM_SIZE }),
   late_ram_regions(Memory_region { }),
   core_mmio(Memory_region { ::Board::UART_BASE, ::Board::UART_SIZE },
             Memory_region { ::Board::Cpu_mmio::IRQ_CONTROLLER_DISTR_BASE,
                             ::Board::Cpu_mmio::IRQ_CONTROLLER_DISTR_SIZE },
             Memory_region { ::Board::Cpu_mmio::IRQ_CONTROLLER_REDIST_BASE,
                             ::Board::Cpu_mmio::IRQ_CONTROLLER_REDIST_SIZE })
 {
     ::Board::Pic pic {};

     ... incomprehensible magic spells, some gibberish about GPIO, CCM, PLL ...
 }

 void Board::Cpu::wake_up_all_cpus(void * ip)
 {
     ... more magic spells, digressing into assembly code ...
 }

The early_ram_regions, late_ram_regions, and core_mmio data structures are initialized with the known ranges of physical memory and memory-mapped I/O registers. This information is designated to be passed further to core.

The call of ::Board::Pic pic {}; performs basic interrupt-controller initialization that is needed only once. It is followed by a sequence of board-specific tweaks to bring the board into a defined state for the kernel to rely on. For instance, setting the I/O MUX configuration, default voltages, and frequencies. The U-boot boot loader already does a fine job for establishing a base line but it is rather conservative. The code for the i.MX8 EVK boosts the voltages and frequencies for improving the performance.

The wake_up_all_cpus call invokes a hook to enable secondary CPU cores. The used mechanism varies from board to board, specifically depending on the operation of the ARM Trusted Firmware. We have to brace ourself for some investigation once we look into multi-processor support. At the beginning, however, we will use only the boot CPU. So we can ignore this function for now.

Finally, let's turn our attention to the core-specific files.

 REP_INC_DIR += src/core/board/imx8q_evk
 REP_INC_DIR += src/core/spec/arm/virtualization

 # add C++ sources
 SRC_CC += kernel/vm_thread_on.cc
 SRC_CC += spec/arm/gicv3.cc
 SRC_CC += spec/arm_v8/virtualization/kernel/vm.cc
 SRC_CC += spec/arm/virtualization/platform_services.cc
 SRC_CC += spec/arm/virtualization/vm_session_component.cc
 SRC_CC += vm_session_common.cc
 SRC_CC += vm_session_component.cc

 #add assembly sources
 SRC_S += spec/arm_v8/virtualization/exception_vector.s

 NR_OF_CPUS = 4

 # include less specific configuration
 include $(call select_from_repositories,lib/mk/spec/arm_v8/core-hw.inc)

Core needs to know the type of the interrupt controller because it processes interrupts at runtime. Here, the GICv3 driver is incorporated.

Similar to bootstrap, a few data structures within core are statically allocated for each CPU, hence the NR_OF_CPUS must be specified here as well.

We can ignore the files with vm_* and virtualization in their names for now. They are important for hosting virtual machines. Since the virtualization support is a generic feature of the ARM CPU, we don't have to take board-specific precautions.

 #include <hw/spec/arm_64/imx8q_evk_board.h>
 #include <spec/arm/generic_timer.h>
 #include <spec/arm/virtualization/gicv3.h>
 #include <spec/arm_v8/cpu.h>
 #include <spec/arm_64/cpu/vm_state_virtualization.h>
 #include <spec/arm/virtualization/board.h>

 namespace Board {
     using namespace Hw::Imx8q_evk_board;

     enum {
         TIMER_IRQ           = 14 + 16,
         VT_TIMER_IRQ        = 11 + 16,
         VT_MAINTAINANCE_IRQ = 9  + 16,
         VCPU_MAX            = 16
     };
 }

In addition to the aggregation of headers matching the board and SoC - like the generic timer driver - we see the definitions of just the few interrupt numbers that are important to core. The kernel is completely oblivious about all other peripheral devices.

The VCPU_MAX definition is solely used for the dimensioning of an array that keeps the state of virtual CPUs for virtual machine. It is not important for now.

A new home for the board support

The easiest way to add support for a new board is the mirroring of the files introduced above. We could march forward with adding new files and directories to a new branch of the Genode repository. Alternatively, the Genode build system allows us to host our custom board-specific files in a dedicated source repository that we can maintain independently from the Genode main repository. The latter approach has the following advantages.

First, it reinforces a clean separation between board-specific code from generic Genode code. In particular, the segregation of code constricts the working set of files relevant for a given board, keeping only important code in view.

Operationally, it allows the decoupling of code ownership in terms of responsibility, quality assurance, licensing hygiene, development process, and the choice of source hosting.

Finally, it alleviates the pressure to agree on one big joint code base, removing potential points of friction between developers.

In the following, we will put our code into a new repository named allwinner.

 mkdir repos/allwinner

In principle, the directory can be anywhere but I find it practical to host it under the repos directory of the Genode source tree. One may also opt to use a symlink, e.g., repos/allwinner pointing to ~/src/genode-allwinner.git.

We need to come up with with a concise name for our board support. Throughout Genode, we follow certain naming conventions. In particular, we use underscore _ for tightly coupled words, and minus - for loosely coupled terms. For example, in the file name core-hw-imx8q_evk.mk, "imx8q_evk" belong closely together whereas the words "core" and "hw" are used as some kind of category (read: the "core" component for the "hw" kernel for the "imx8q_evk" board). With the background of these conventions, the board name pine_a64lts seems sensible. Specific enough while still concise.

For the initial content from our new allwinner repository be blatantly mirror the files of the base-hw repository.

 $ mkdir -p allwinner/src/include/hw/spec/arm_64/
 $ cp imx/src/include/hw/spec/arm_64/imx8q_evk_board.h \
      allwinner/src/include/hw/spec/arm_64/pine_a64lts_board.h
 $ mkdir -p allwinner/lib/mk/spec/arm_v8
 $ cp imx/lib/mk/spec/arm_v8/bootstrap-hw-imx8q_evk.mk \
      allwinner/lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk
 $ mkdir -p allwinner/src/bootstrap/board/pine_a64lts
 $ cp imx/src/bootstrap/board/imx8q_evk/board.h \
      allwinner/src/bootstrap/board/pine_a64lts/
 $ cp imx/src/bootstrap/board/imx8q_evk/platform.cc \
      allwinner/src/bootstrap/board/pine_a64lts/
 $ cp imx/lib/mk/spec/arm_v8/core-hw-imx8q_evk.mk \
      allwinner/lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk
 $ mkdir -p allwinner/src/core/board/pine_a64lts
 $ cp imx/src/core/board/imx8q_evk/board.h \
      allwinner/src/core/board/pine_a64lts/

At the current stage, we are concerned about getting the build process right. To concentrate at this one thing at a time, let us pretend that the Pine-A64-LTS board works equal to the i.MX8 EVK. We don't mind that the technicalities copied from the existing board don't match our new board until we run the code on the board. That said, as the build-description files (those with the mk suffix) steer the build process, they must be made consistent with our directory structure. So we have to revisit those files while looking out for the pattern imx8q_evk.

A look into lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk reveals the following line:

 REP_INC_DIR += src/bootstrap/board/imx8q_evk

We have to replace it with

 REP_INC_DIR += src/bootstrap/board/pine_a64lts

Similarly, allwinner/lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk contains the line:

 REP_INC_DIR += src/core/board/imx8q_evk

This must be changed to

 REP_INC_DIR += src/core/board/pine_a64lts

System-integration dry-run

Let us see how the Genode build system swallows - or chokes on - our new board support. First, we need a build directory for the ARMv8 architecture.

 $ ./tool/create_builddir arm_v8a
 Successfully created build directory at /.../genode/build/arm_v8a.
 Please adjust /.../genode/build/arm_v8a/etc/build.conf according to your needs.

As suggested, we open build/etc/build.conf in our favorite text editor. Normally, I enable parallel builds by uncommenting the corresponding line right at the beginning of the file. But for now, let us keep it disabled until the skeleton builds successfully. The steps of the build system are easier to follow if it operates deterministically.

We need to extend the REPOSITORIES variable with the path to our custom repository. For the allwinner repository, that would be following line:

 REPOSITORIES += $(GENODE_DIR)/repos/allwinner

Note that the order of REPOSITORIES defines the search order of the build system for files. If the allwinner repository should be able to override content of the other repositories, specifically base-hw, the above line should appear before the others.

With these changes in place, we can issue the build of bootstrap for new board.

 $ cd build/arm_v8a
 $ make bootstrap/hw KERNEL=hw BOARD=pine_a64lts
  ...
  Library bootstrap-hw-pine_a64lts
    ...
    MERGE    bootstrap-hw-pine_a64lts.lib.a
  Program bootstrap/hw/bootstrap_hw_pine_a64lts

The result can be found in the sub directory bootstrap/hw/. We find a single object file named bootstrap-hw-pine_a64lts.o along with a stripped version of this file.

Likewise, core for the base-hw kernel and the new board can be built as follows.

 $ make core KERNEL=hw BOARD=pine_a64lts
    ...
    MERGE    core-hw-pine_a64lts.lib.a
  Program core/hw/core_hw_pine_a64lts

Similar to the build of bootstrap, we can find the result at the corresponding subdirectory, here core/hw/. We find a single archive file named core-hw-pine_a64lts.a along with a stripped version of this file.

Next up, we are going to build a system image that contains both core and bootstrap. Now would be a good time to enable parallel builds. Edit the etc/build.conf file by un-commenting the following line (removing the hash # character).

 #MAKE += -j4

One may also opt to write the BOARD and KERNEL arguments directly into the build.conf file as illustrated by the commented-out examples. This spares the need to specify the arguments each time when issuing a build command.

A system image contains bootstrap, core, and additional boot modules. The first two puzzle pieces are already in place. But what about the boot modules? In contrast to bootstrap and core, which are always the same for each system scenario, the boot modules vary between system scenarios. Genode system scenarios are defined in the form of run scripts. The run script at repos/base/run/log.run is a good starting point. As defined by this particular run script, the system image for the "log" system scenario is comprised of core, init, ld.lib.so, init, and test-log in addition to a configuration. A system image (image.elf) for this scenario would look like this:

Genode's run tool automates the process of assembling such Matryoshkas from the various pieces. Let's give it a try:

 $ make run/log KERNEL=hw BOARD=pine_a64lts
 ...
 ... long sequence of compile steps
 ...
 genode build completed
 using 'ld-hw.lib.so' as 'ld.lib.so'
 core link address is 0xffffffc000000000

 Error: unknown image link address

  File board/pine_a64lts/image_link_address not present in any repository.

 Makefile:329: recipe for target 'run/log' failed

This message should prompt us to have closer look at the run tool.

 $ cd genode
 $ grep -r "unknown image link address" tool
 tool/run/boot_dir/hw: puts stderr "\nError: unknown image link address\n"

The file tool/run/boot_dir/hw is the part of the run tool that defines the integration of a system image from its parts for the base-hw kernel. It is worth skimming over the file to get a rough understanding of how the system image is assembled from its ingredients. The error message above comes from the function bootstrap_link_address called during the system-image integration step.

The link address is evaluated by the boot loader when loading the system image as ELF binary. It defines the start of the text segment of the system image in physical memory. As the physical memory layout differs between SoCs and boards, we must provide a value that is suitable for the memory layout of the Pine-A64-LTS board. From looking at Linux' /proc/iomem, we remember that the system RAM of our board starts at 0x40000000.

As indicated by the error message above, the run tool expects to find the link address in a file called board/pine_a64lts/image_link_address. Let's create such a file with a sensible value. It is common practice to leave some room at the very beginning of the memory, which is often occupied by the boot loader. It is usually fine to link the system image to 64 KiB after the start of the physical memory.

 $ cd allwinner
 $ mkdir -p board/pine_a64lts
 $ echo 0x40010000 > board/pine_a64lts/image_link_address

With the link address defined, another attempt to build the system image for the log scenario succeeds. The result can be found in the build directory's var/run/ sub directory:

 $ find var/run
 var/run
 var/run/log.boot_modules.o
 var/run/log
 var/run/log/boot
 var/run/log/boot/image.elf
 var/run/log.core
 var/run/log.bootstrap
 var/run/log.config

The most interesting file is certainly var/run/log/boot/image.elf, which is the final system image. To quickly validate the link address, let's check the ELF entrypoint.

 $ readelf -a var/run/log/boot/image.elf | grep Entry
   Entry point address:               0x40010000

The value looks familiar. While we are at it, the other files are also worth inspecting.

By the way, one may prefer booting a uImage instead of an ELF image because a uImage is compressed using gzip by default, which reduces the boot time. The run tool supports that via the argument --include image/uboot. One can either extend the RUN_OPT variable by adding a corresponding line to etc/build.conf or pass the option to the make command line:

 $ RUN_OPT='--include image/uboot' make run/log BOARD=pine_a64lts KERNEL=hw

After completing the build, the uImage file can be found at var/run/log/uImage.

This is not magic. At this point, I recommend taking a look at the run tool's snippets located at tool/run/. In particular, tool/run/image/uboot contains the sequence of commands used for generating the uImage from the ELF image.

Getting to grips using meaningful numbers

The faux system image that we just created contains information cowardly copied from the imx8q_evk board, and which certainly mismatches the pine_a64lts board. So let's revisit the files in our repository one by one and look out for any numbers. Numbers are important. According to my experience, hexadecimal numbers are especially important. Don't forget to squinch your eyes when looking at them. Change them with caution.

 $ cd repos/allwinner
 $ find -type f
 ./lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk
 ./lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk
 ./board/pine_a64lts/image_link_address
 ./src/bootstrap/board/pine_a64lts/platform.cc
 ./src/bootstrap/board/pine_a64lts/board.h
 ./src/include/hw/spec/arm_64/pine_a64lts_board.h
 ./src/core/board/pine_a64lts/board.h

The following line catches our attention:

 SRC_CC  += bootstrap/spec/arm/gicv3.cc

The i.MX8 SoC uses ARM's Generic Interrupt Controller version 3 (GICv3). From booting Linux on the Pine-A64 board, we learned that the Allwinner SoC uses the GIC version 2. Fortunately, the base-hw kernel supports both versions. So we can change the line to:

 SRC_CC  += bootstrap/spec/arm/gicv2.cc

The NR_OF_CPUS value can stay unmodified because the Allwinner SoC has 4 cores.

We merely also have to adjust the GIC version from 3 to 2.

The file contains a lot of i.MX8Q-specific initialization steps like tweaking clocks and voltages. We can remove this code without looking back. The body of the Bootstrap::Platform::Board constructor can be reduced to the mere initialization of the interrupt controller:

 {
   ::Board::Pic pic { };
 }

The list of memory regions passed to the core_mmio member can be pruned to the single entry for the UART. The other entries that refer to the IRQ controller should be removed because they refer to the wrong version of the GIC anyway. We will supplement the proper regions for the GICv2 later, once we turn our attention to interrupts.

 core_mmio(Memory_region { ::Board::UART_BASE, ::Board::UART_SIZE })

At this point, I am admittedly unsure about the wake_up_all_cpus implementation, in particular whether the opcode of the CPU_ON smc instruction would match. I guess not. We will come to multi-processor support at a later stage. So let's better remove the uncertainty by reducing the implementation to

 void Board::Cpu::wake_up_all_cpus(void *) { }

We see several things that cry for adjustment.

• Updating the include guards

• Including the correct board definitions by replacing

   #include <hw/spec/arm_64/imx8q_evk_board.h>
  

  by

   #include <hw/spec/arm_64/pine_a64lts_board.h>
  

• Incorporating the GICv2 driver instead of the GICv3 driver by changing

   #include <hw/spec/arm/gicv3.h>
  

  to

   #include <hw/spec/arm/gicv2.h>
  

• Defining the C++ type Pic such that it refers to the Hw::Gicv2 driver:

   using Pic = Hw::Gicv2;
  

To our despair, the file is full of numbers.

• It includes the driver for the UART used for printing debug messages. Of course, the specified drivers/uart/imx.h driver won't work. While experimenting with the bare-metal serial output, we have learned that the Allwinner SoC uses a NS16550 UART controller. Let us pretend having a driver by changing the line to

   #include <drivers/uart/ns16550.h>
  

• The board-specific name space should reflect the name of our board:

   namespace Hw::Pine_a64lts_board {
  

• We want the C++ type Hw::Serial to refer to our hypothetical NS16550 driver.

   using Serial = Genode::Ns16550_uart;
  

• The RAM_BASE and RAM_SIZE values must match those we found from the look at Linux /proc/iomem.

   RAM_BASE = 0x40000000,
   RAM_SIZE = 0x7e000000,
  

• We already have found known-good values for UART_BASE and UART_SIZE during our bare-metal serial output experimentation. The UART_CLOCK value won't be needed in our case. So we define it as zero.

   UART_BASE  = 0x1c28000,
   UART_SIZE  = 0x1000,
   UART_CLOCK = 0,
  

• The IRQ_CONTROLLER_REDIST_BASE and SIZE are not used for the GICv2. So the values can be removed.

• The values for IRQ_CONTROLLER_DISTR_BASE and SIZE as well as VT_CPU_BASE and SIZE will become important once we will turn our attention to the interrupt controller. But this is not today. So we keep the existing numbers, keeping in mind that they won't work.

• When using the GICv2, we need to add the definition of IRQ_CONTROLLER_CPU_BASE and VT_CTRL_BASE. Until we use interrupts, we can pick an arbitrary number. To display good manners, let's leave the lowest 12 bits to zero, pretending that each device resource starts at a page boundary.

   IRQ_CONTROLLER_CPU_BASE     = 0xaaaaa000,
   IRQ_CONTROLLER_VT_CTRL_BASE = 0xbbbbb000,
  

The file contains mostly interrupt numbers. We will turn our attention to interrupts later. Let's not touch them for now because we cannot validate the values anyway at this point. Apart from these numbers, a few adjustments must be made.

• Updating the include guard

• Including the board definitions from pine_a64lts_board.h

• Adjusting the GIC version of the included header from gicv3.h to gicv2.h

• Importing the board-specific namespace Hw::Pine_a64lts_board

To wrap up this step, let's check if we missed any leftover by grepping for remaining occurrences of patterns like "imx" or "gicv3".

 $ grep -ri imx repos/allwinner

Now would also be a good time to revisit the file headers, updating the information about the author, creation date, brief description, and copyright. Should the code be considered to eventually become part of the Genode upstream project at some point, it is sensible to leave the license disclaimer as is, clarifying that the code is designated be a part of the Genode OS framework.

UART driver for bootstrap and core

The next attempt to build the system image for the log scenario fails predictably:

 $ make run/log KERNEL=hw BOARD=pine_a64lts
 ...
     COMPILE  core_region_map.o
 In file included from /.../repos/allwinner/src/core/board/pine_a64lts/board.h:17,
                  from /.../repos/base-hw/src/core/platform.h:37,
                  from /.../repos/base-hw/src/core/core_region_map.cc:18:
 /.../repos/allwinner/src/include/hw/spec/arm_64/pine_a64lts_board.h:17:10:
            fatal error: drivers/uart/ns16550.h: No such file or directory
  #include <drivers/uart/ns16550.h>
           ^~~~~~~~~~~~~~~~~~~~~~~~

We can find a number of blueprints for our new UART driver at repos/base/include/drivers/uart/. By following the lines of the existing drivers and combining our knowledge from the bare-metal serial experiments, we can come up with the following little driver placed at allwinner/include/drivers/uart/ns16550.h.

 #include <util/mmio.h>

 namespace Genode { class Ns16550_uart; }


 class Genode::Ns16550_uart : Mmio
 {
   private:

     struct Thr : Register<0x00, 32>
     {
       struct Data : Bitfield<0,8> { };
     };

     struct Lsr : Register<0x14, 32>
     {
       struct Thr_empty : Bitfield<5,1> { };
     };

   public:

     Ns16550_uart(addr_t const base, uint32_t, uint32_t) : Mmio(base) { }

     void put_char(char const c)
     {
       while (read<Lsr::Thr_empty>() == 0);

       write<Thr::Data>(c);
     }
 };

Like all drivers dedicatedly developed for Genode, it uses Genode's Register API to safely access bits of memory-mapped I/O registers. You can find the API described in Section Utilities for user-level device drivers of the Genode-Foundations book.

Climbing the mountain step by step

We are almost there. On our walk, we repeatedly try to build the system image, look at the compiler and linker errors, fix them, and repeat.

 $ make run/log KERNEL=hw BOARD=pine_a64lts
     ...
     COMPILE  bootstrap/spec/arm/gicv2.o
 /.../repos/base-hw/src/bootstrap/spec/arm/gicv2.cc:
                     In constructor 'Hw::Gicv2::Gicv2()':
 /.../repos/base-hw/src/bootstrap/spec/arm/gicv2.cc:23:28:
                     error: 'NON_SECURE' is not a member of 'Board'
   bool use_group_1 = Board::NON_SECURE &&
                             ^~~~~~~~~~

The interrupt-controller driver apparently needs to distinguish the cases where the kernel is running in the so-called "secure world" or "normal world" of ARM TrustZone. If you want to learn more about schizophrenia as a feature of ARM processors, let me point you to our article on ARM TrustZone. Admittedly, I'm not completely sure about which of both worlds are executing our kernel. But it is probably safe to assume that the boot process switches to the normal world before loading and starting our system image. So we add the definition of NON_SECURE to allwinner/src/bootstrap/board/pine_a64lts/board.h.

 namespace Board {
   ...
   static constexpr bool NON_SECURE = true;
 }

The next slope on our way up the hill:

 $ make run/log KERNEL=hw BOARD=pine_a64lts
     ...
     MERGE    bootstrap-hw-pine_a64lts.lib.a
 /.../genode-aarch64-ar: bootstrap/board/pine_a64lts/platform.o:
                         No such file or directory
 /.../repos/base/mk/lib.mk:180: recipe for target
                                'bootstrap-hw-pine_a64lts.lib.a' failed

We have to guide the build system to consider source files in the allwinner repository, by adding the following line to lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk.

 vpath bootstrap/% $(REP_DIR)/src

Next try. This time, we get a link error:

 /.../aarch64-none-elf/bin/ld: debug/core-hw-pine_a64lts.a(cpu.o):
                               in function `Board::Pic::Pic()':
 /.../repos/base-hw/src/core/spec/arm/virtualization/gicv2.h:22:
     undefined reference to `Board::Pic::Gich::Gich()'

It turns out that the virtualization-related parts of the GICv2 driver reside in a distinct compilation unit located at base-hw/src/core/spec/arm/virtualization/gicv2.cc, which is not yet included in the build description for core. We have to add the following line to allwinner/lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk.

 SRC_CC += spec/arm/virtualization/gicv2.cc

With these minor obstacles addressed, we get a system image that should largely be compatible with our board. The urge to try out the freshly baked system image on the board is strong. Why not?

Looking from the top of the world

Testing the system image on the board comes down to the following few steps.

1. Make sure to build the uImage using the image/uboot RUN_OPT.

    $ RUN_OPT='--include image/uboot' make run/log BOARD=pine_a64lts KERNEL=hw
   

2. Copy the uImage from build/arm_v8a/var/run/log/uImage to the TFTP directory. In my case, that is /var/lib/tftpboot/.

3. Boot the board and use U-Boot's bootp and bootm commands to load the uImage via TFTP and start it.

 => bootp 10.0.0.32:/var/lib/tftpboot/uImage
 BOOTP broadcast 1
 BOOTP broadcast 2
 BOOTP broadcast 3
 DHCP client bound to address 10.0.0.178 (1121 ms)
 Using ethernet@1c30000 device
 TFTP from server 10.0.0.32; our IP address is 10.0.0.178
 Filename '/var/lib/tftpboot/uImage'.
 Load address: 0x42000000
 Loading: #############################################################
          2.7 MiB/s
 done
 => bootm
 ## Booting kernel from Legacy Image at 42000000 ...
    Image Name:   
    Image Type:   AArch64 Linux Kernel Image (gzip compressed)
    Data Size:    887610 Bytes = 866.8 KiB
    Load Address: 40010000
    Entry Point:  40010000
    Verifying Checksum ... OK
    Uncompressing Kernel Image

 Starting kernel ...

 Error: Assertion failed: id < _count && _cpus[id].constructed()
 Error:   File: /.../repos/base-hw/src/core/kernel/cpu.cc:205
 Error:   Function: Kernel::Cpu& Kernel::Cpu_pool::cpu(unsigned int)

The excitement is real! That's the first life sign of Genode's kernel! We get three satisfactory results at once. First, our custom Ns16550_uart driver is working, as evidenced by the beautifully formatted error messages. So we did not mess up any of the important numbers there. Second, in contrast to the archaic experiments with the bare-metal serial output, which did not even use a stack, we can now enjoy the comfort of Genode's C++ runtime. We don't feel like living in a cave any longer. Finally, we got a meaningful error message, which is an unmistakable direction sign for our next steps.