Yocto for Adaptive Autosar

The Yocto Project is a set of templates, tools and methods that allow to build custom embedded Linux-based systems.

The core components of the Yocto Project are:
  - BitBake, the build engine. It is a task scheduler, like make. It interprets configuration files and recipes (also called metadata) to perform a set of tasks, to download, configure and build specified applications and filesystem images.
  - OpenEmbedded-Core, a set of base layers. It is a set of recipes, layers and classes which are shared between all OpenEmbedded based systems.
  - Poky, the reference system. It is a collection of projects and tools, used to bootstrap a new distribution based on the Yocto Project.

To build images for a BeagleBone Black, we need:
  - The Poky reference system, containing all common recipes and tools.
  - The meta-ti layer, a set of Texas Instruments specific recipes.
  - All modifications are made in the meta-ti layer. Editing Poky is a no-go!

To download the Poky reference system:
git clone -b sumo git://git.yoctoproject.org/ poky.git

The poky/ directory
  bitbake              =>   Holds all scripts used by the BitBake command. Usually matches the stable release of the BitBake project.
  meta                 =>   Contains the OpenEmbedded-Core metadata.
  meta-skeleton        =>   Contains template recipes for BSP and kernel development. BSP work: porting the bootloader and Linux kernel, developing Linux device drivers.
  meta-poky            =>   Holds the configuration for the Poky reference distribution.
  meta-yocto-bsp       =>   Configuration for the Yocto Project reference hardware board support package.
  oe-init-build-env    =>  Script to set up the OpenEmbedded build environment. It will create the build directory. It takes an optional parameter which is the build directory name. By default, this is build. This script has to be sourced because it changes environment variables.
  scripts              =>  Contains scripts used to set up the environment, development tools, and tools to flash the generated images on the target.

Using command "source ./oe-init-build-env [builddir]"
  => If not provided, the default name is build

The build/ directory
  conf                    =>      Configuration files. Image specific and layer configuration.
  downloads               =>      Downloaded upstream tarballs of the recipes used in the builds.
  sstate-cache            =>      Shared state cache. Used by all builds.
  tmp                     =>      Holds all the build system outputs.
  tmp/buildstats          =>      Build statistics for all packages built (CPU usage, elapsed time, host, timestamps...).
  tmp/deploy              =>      Final output of the build.
  tmp/deploy/images       =>      Contains the complete images built by the OpenEmbedded build system. These images are used to flash the target.
  tmp/work                =>      Set of specific work directories, split by architecture. They are used to unpack, configure and build the packages. Contains the patched sources, generated objects and logs.
  tmp/sysroots            =>      Shared libraries and headers used to compile applications for the target but also for the host.

Configuring the build system
  - The build/conf/ directory
  - bblayers.conf       =>  Explicitly list the available layers.
  - local.conf          =>  Set up the configuration variables relative to the current user for the build. Configuration variables can be overridden there.

The conf/local.conf =>  configuration file holds local user configuration variables:
    BB_NUMBER_THREADS =>  How many tasks BitBake should perform in parallel. Defaults to the number of CPUs on the system (e.g: BB_NUMBER_THREADS ?= "4")
    PARALLEL_MAKE =>  How many processes should be used when compiling. Defaults to the number of CPUs on the system (e.g: PARALLEL_MAKE ?= "-j 4")
    MACHINE =>  The machine the target is built for, e.g. beaglebone.

Common targets are listed when sourcing the script:
  core-image-minimal  =>  A small image to boot a device and have access to core command line commands and services.
  core-image-sato     =>  Image with Sato support. Sato is a GNOME mobile-based user interface.
  meta-toolchain      =>  Includes development headers and libraries to develop directly on the target.
  meta-ide-support    =>  Generates the cross-toolchain. Useful when working with the SDK.
 
Building an image
  - To build a target: bitbake [target]
    Example: Building a minimal image: bitbake core-image-minimal
  - More options:
    -c <task> execute the given task
    -s list all locally available packages and their versions
    -f force the given task to be run by removing its stamp file world keyword for all recipes
    -b <recipe> execute tasks from the given recipe (without resolving dependencies).
    Examples:
      bitbake -c listtasks virtual/kernel  => Gives a list of the available tasks for the recipe providing the package virtual/kernel. Tasks are prefixed with do_.
      bitbake -c menuconfig virtual/kernel => Execute the task menuconfig on the recipe providing the virtual/kernel package.
      bitbake -f dropbear => Force the dropbear recipe to run all tasks.
      bitbake world --runall=fetch => Download all recipe sources and their dependencies.
      For a full description: bitbake --help
  - BitBake stores the output of each task in a directory, the shared state cache. Its location is controlled by the SSTATE_DIR variable. It is possible to clean old data with: ./scripts/sstate-cache-management.sh --remove-duplicated -d --cache-dir=<SSTATE_DIR>
 
Advanced configuration
  The OpenEmbedded build system uses configuration variables to hold information.
  Configuration settings are in upper-case by convention, e.g. CONF_VERSION
  To make configuration easier, it is possible to prepend, append or define these variables in a conditional way.
  All variables can be overridden or modified in $BUILDDIR/conf/local.conf

If we define:
  IMAGE_INSTALL = "busybox mtd-utils"
  IMAGE_INSTALL_append = " dropbear"
  IMAGE_INSTALL_append_beaglebone = " i2c-tools"
The resulting configuration variable will be:
  IMAGE_INSTALL = "busybox mtd-utils dropbear i2c-tools" if the machinebeing built is beaglebone.
  Otherwise:
  IMAGE_INSTALL = "busybox mtd-utils dropbear"

The most specific variable takes precedence.
  IMAGE_INSTALL_beaglebone = "busybox mtd-utils i2c-tools"
  IMAGE_INSTALL = "busybox mtd-utils"
If the machine is beaglebone:
  IMAGE_INSTALL = "busybox mtd-utils i2c-tools"
  Otherwise:
  IMAGE_INSTALL = "busybox mtd-utils"
 
Operators can be used to assign values to configuration variables:
  = expand the value when using the variable
  := immediately expand the value
  += append (with space)
  =+ prepend (with space)
  .= append (without space)
  =. prepend (without space)
  ?= assign if no other value was previously assigned
  ??= same as previous, with a lower precedence 
 
Packages variants
  - The build system uses virtual packages (form virtual/<name>) to describe functionalities and several packages may provide it. And only one can be used at a time.
  Examples:
    virtual/bootloader: u-boot, u-boot-ti-staging...
    virtual/kernel: linux-yocto, linux-yocto-tiny, linux-yocto-rt, linux-ti-staging...
  - Specify a package by PREFERRED_PROVIDER : PREFERRED_PROVIDER_virtual/kernel ?= "linux-ti-staging"
    or PREFERRED_VERSION_linux-yocto = "3.10\%" (% is wildcard)
 
Packages
  - Packages are controlled by the IMAGE_INSTALL configuration variable.
  - The list of packages to install is also filtered using the PACKAGE_EXCLUDE variable. However, if a package needs installing to satisfy a dependency, it will still be
selected.
  - The set of packages installed into the image is defined by the target you choose (e.g. core-image-minimal).

Recipes
  - A recipe is a set of instructions to describe how to retrieve, patch, compile, install and generate binary packages for a given application.
  - It also defines what build or runtime dependencies are required.
  - A recipe contains configuration variables: name, license, dependencies, path to retrieve the source code...
  - It also contains functions that can be run (fetch, configure, compile...) which are called tasks.
  - Tasks provide a set of actions to perform.
  - The recipes are parsed by the BitBake build engine.
  - The format of a recipe file name is <application-name>_<version>.bb
  - Many applications have more than one recipe, to support different versions. In that case the common metadata is included in each version specific recipe and is in a .inc file:
      <application>.inc: version agnostic metadata.
      <application>_<version>.bb: require <application>.inc and version specific metadata.
  - We can divide a recipe into three main parts:
      + The header: what/who
      + The sources: where
      + The tasks: how

Practice notes:
1. Build Yocto with quemux86
cd ~/raspberrypi0_wifi
git clone http://git.yoctoproject.org/cgit.cgi/poky/
source poky/oe-init-build-env build
bitbake core-image-satorunqemu qemux86
2. Build a yoto raspberry pi0 with wifi
Commands:
cd poky
git clone git://git.openembedded.org/meta-openembedded
git clone git://git.yoctoproject.org/meta-raspberrypi
In  "local.conf" change:
--------------------
  # MACHINE ??= "qemux86"
  MACHINE ??= "raspberrypi0-wifi" # look in "poky/meta-raspberrypi/conf/machine"
--------------------
In "bblayers.conf" change:
--------------------
BBLAYERS ?= " \
/home/<user>/yoctoproject/poky/meta \
/home/<user>/yoctoproject/poky//meta-poky \
/home/<user>/yoctoproject/poky//meta-yocto-bsp \
    "
--------------------
BBLAYERS ?= " \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-poky \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-yocto-bsp \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-oe \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-multimedia \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-networking \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-python \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-filesystems \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-gnome \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-xfce \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-raspberrypi \
    "
--------------------
or using commands to Add layers from command line:
source poky/oe-init-build-env
bitbake-layers add-layer ../your_layers

Build an image (check avialable receipes in meta-raspberrypi/recipes-core/images):
bitbake rpi-hwup-image

After finishing building process write image to sdcard of Raspberry:
sudo dd if=./tmp/deploy/images/raspberrypi/rpi-hwup-image-raspberrypi.rpi-sdimg of=/dev/mmcblk0

How to install apt-get and packages in Yocto:
1. In "build/conf/local.conf" add these lines:
PACKAGE_CLASSES = "package_deb"
PACKAGE_FEED_URIS = "http://<ip-of-machine-host-packages>:5678"
EXTRA_IMAGE_FEATURES += " package-management "
2. Build the image (e.g: core-image-sato):
bitbake core-image-sato
3. Create the package index:
bitbake package-index
4. Start packages server so the client can download packages:
cd build/tmp/deploy/deb
python -m SimpleHTTPServer 5678
5. In target side (target that run image), update apt and install:
apt-get update
6. In order to install new packages (e.g: cmake):
- On server side:
bitbake cmake
bitbake package-index
- On target side:
apt-get update
apt-cache search cmake
apt-get install cmake

Clean image when changing recipes:
- Yocto is not designed for development but rather for distribution it won’t strictly follow up on files being changed. You would need to clean and then bake again so make sure that the changes are reflected. Command:
bitbake –c clean <RECIPE OR IMAGE>
bitbake < RECIPE OR IMAGE >