Yocto for Adaptive Autosar

The Yocto Project is a set of templates, tools and methods that allow to build custom embedded Linux-based systems.

The core components of the Yocto Project are:
  - BitBake, the build engine. It is a task scheduler, like make. It interprets configuration files and recipes (also called metadata) to perform a set of tasks, to download, configure and build specified applications and filesystem images.
  - OpenEmbedded-Core, a set of base layers. It is a set of recipes, layers and classes which are shared between all OpenEmbedded based systems.
  - Poky, the reference system. It is a collection of projects and tools, used to bootstrap a new distribution based on the Yocto Project.

To build images for a BeagleBone Black, we need:
  - The Poky reference system, containing all common recipes and tools.
  - The meta-ti layer, a set of Texas Instruments specific recipes.
  - All modifications are made in the meta-ti layer. Editing Poky is a no-go!

To download the Poky reference system:
git clone -b sumo git://git.yoctoproject.org/ poky.git

The poky/ directory
  bitbake              =>   Holds all scripts used by the BitBake command. Usually matches the stable release of the BitBake project.
  meta                 =>   Contains the OpenEmbedded-Core metadata.
  meta-skeleton        =>   Contains template recipes for BSP and kernel development. BSP work: porting the bootloader and Linux kernel, developing Linux device drivers.
  meta-poky            =>   Holds the configuration for the Poky reference distribution.
  meta-yocto-bsp       =>   Configuration for the Yocto Project reference hardware board support package.
  oe-init-build-env    =>  Script to set up the OpenEmbedded build environment. It will create the build directory. It takes an optional parameter which is the build directory name. By default, this is build. This script has to be sourced because it changes environment variables.
  scripts              =>  Contains scripts used to set up the environment, development tools, and tools to flash the generated images on the target.

Using command "source ./oe-init-build-env [builddir]"
  => If not provided, the default name is build

The build/ directory
  conf                    =>      Configuration files. Image specific and layer configuration.
  downloads               =>      Downloaded upstream tarballs of the recipes used in the builds.
  sstate-cache            =>      Shared state cache. Used by all builds.
  tmp                     =>      Holds all the build system outputs.
  tmp/buildstats          =>      Build statistics for all packages built (CPU usage, elapsed time, host, timestamps...).
  tmp/deploy              =>      Final output of the build.
  tmp/deploy/images       =>      Contains the complete images built by the OpenEmbedded build system. These images are used to flash the target.
  tmp/work                =>      Set of specific work directories, split by architecture. They are used to unpack, configure and build the packages. Contains the patched sources, generated objects and logs.
  tmp/sysroots            =>      Shared libraries and headers used to compile applications for the target but also for the host.

Configuring the build system
  - The build/conf/ directory
  - bblayers.conf       =>  Explicitly list the available layers.
  - local.conf          =>  Set up the configuration variables relative to the current user for the build. Configuration variables can be overridden there.

The conf/local.conf =>  configuration file holds local user configuration variables:
    BB_NUMBER_THREADS =>  How many tasks BitBake should perform in parallel. Defaults to the number of CPUs on the system (e.g: BB_NUMBER_THREADS ?= "4")
    PARALLEL_MAKE =>  How many processes should be used when compiling. Defaults to the number of CPUs on the system (e.g: PARALLEL_MAKE ?= "-j 4")
    MACHINE =>  The machine the target is built for, e.g. beaglebone.

Common targets are listed when sourcing the script:
  core-image-minimal  =>  A small image to boot a device and have access to core command line commands and services.
  core-image-sato     =>  Image with Sato support. Sato is a GNOME mobile-based user interface.
  meta-toolchain      =>  Includes development headers and libraries to develop directly on the target.
  meta-ide-support    =>  Generates the cross-toolchain. Useful when working with the SDK.
 
Building an image
  - To build a target: bitbake [target]
    Example: Building a minimal image: bitbake core-image-minimal
  - More options:
    -c <task> execute the given task
    -s list all locally available packages and their versions
    -f force the given task to be run by removing its stamp file world keyword for all recipes
    -b <recipe> execute tasks from the given recipe (without resolving dependencies).
    Examples:
      bitbake -c listtasks virtual/kernel  => Gives a list of the available tasks for the recipe providing the package virtual/kernel. Tasks are prefixed with do_.
      bitbake -c menuconfig virtual/kernel => Execute the task menuconfig on the recipe providing the virtual/kernel package.
      bitbake -f dropbear => Force the dropbear recipe to run all tasks.
      bitbake world --runall=fetch => Download all recipe sources and their dependencies.
      For a full description: bitbake --help
  - BitBake stores the output of each task in a directory, the shared state cache. Its location is controlled by the SSTATE_DIR variable. It is possible to clean old data with: ./scripts/sstate-cache-management.sh --remove-duplicated -d --cache-dir=<SSTATE_DIR>
 
Advanced configuration
  The OpenEmbedded build system uses configuration variables to hold information.
  Configuration settings are in upper-case by convention, e.g. CONF_VERSION
  To make configuration easier, it is possible to prepend, append or define these variables in a conditional way.
  All variables can be overridden or modified in $BUILDDIR/conf/local.conf

If we define:
  IMAGE_INSTALL = "busybox mtd-utils"
  IMAGE_INSTALL_append = " dropbear"
  IMAGE_INSTALL_append_beaglebone = " i2c-tools"
The resulting configuration variable will be:
  IMAGE_INSTALL = "busybox mtd-utils dropbear i2c-tools" if the machinebeing built is beaglebone.
  Otherwise:
  IMAGE_INSTALL = "busybox mtd-utils dropbear"

The most specific variable takes precedence.
  IMAGE_INSTALL_beaglebone = "busybox mtd-utils i2c-tools"
  IMAGE_INSTALL = "busybox mtd-utils"
If the machine is beaglebone:
  IMAGE_INSTALL = "busybox mtd-utils i2c-tools"
  Otherwise:
  IMAGE_INSTALL = "busybox mtd-utils"
 
Operators can be used to assign values to configuration variables:
  = expand the value when using the variable
  := immediately expand the value
  += append (with space)
  =+ prepend (with space)
  .= append (without space)
  =. prepend (without space)
  ?= assign if no other value was previously assigned
  ??= same as previous, with a lower precedence 
 
Packages variants
  - The build system uses virtual packages (form virtual/<name>) to describe functionalities and several packages may provide it. And only one can be used at a time.
  Examples:
    virtual/bootloader: u-boot, u-boot-ti-staging...
    virtual/kernel: linux-yocto, linux-yocto-tiny, linux-yocto-rt, linux-ti-staging...
  - Specify a package by PREFERRED_PROVIDER : PREFERRED_PROVIDER_virtual/kernel ?= "linux-ti-staging"
    or PREFERRED_VERSION_linux-yocto = "3.10\%" (% is wildcard)
 
Packages
  - Packages are controlled by the IMAGE_INSTALL configuration variable.
  - The list of packages to install is also filtered using the PACKAGE_EXCLUDE variable. However, if a package needs installing to satisfy a dependency, it will still be
selected.
  - The set of packages installed into the image is defined by the target you choose (e.g. core-image-minimal).

Recipes
  - A recipe is a set of instructions to describe how to retrieve, patch, compile, install and generate binary packages for a given application.
  - It also defines what build or runtime dependencies are required.
  - A recipe contains configuration variables: name, license, dependencies, path to retrieve the source code...
  - It also contains functions that can be run (fetch, configure, compile...) which are called tasks.
  - Tasks provide a set of actions to perform.
  - The recipes are parsed by the BitBake build engine.
  - The format of a recipe file name is <application-name>_<version>.bb
  - Many applications have more than one recipe, to support different versions. In that case the common metadata is included in each version specific recipe and is in a .inc file:
      <application>.inc: version agnostic metadata.
      <application>_<version>.bb: require <application>.inc and version specific metadata.
  - We can divide a recipe into three main parts:
      + The header: what/who
      + The sources: where
      + The tasks: how

Practice notes:
1. Build Yocto with quemux86
cd ~/raspberrypi0_wifi
git clone http://git.yoctoproject.org/cgit.cgi/poky/
source poky/oe-init-build-env build
bitbake core-image-satorunqemu qemux86
2. Build a yoto raspberry pi0 with wifi
Commands:
cd poky
git clone git://git.openembedded.org/meta-openembedded
git clone git://git.yoctoproject.org/meta-raspberrypi
In  "local.conf" change:
--------------------
  # MACHINE ??= "qemux86"
  MACHINE ??= "raspberrypi0-wifi" # look in "poky/meta-raspberrypi/conf/machine"
--------------------
In "bblayers.conf" change:
--------------------
BBLAYERS ?= " \
/home/<user>/yoctoproject/poky/meta \
/home/<user>/yoctoproject/poky//meta-poky \
/home/<user>/yoctoproject/poky//meta-yocto-bsp \
    "
--------------------
BBLAYERS ?= " \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-poky \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-yocto-bsp \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-oe \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-multimedia \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-networking \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-python \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-filesystems \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-gnome \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-openembedded/meta-xfce \
/home/tuan/working/learning/autosar/yocto/test_rpi/poky/meta-raspberrypi \
    "
--------------------
or using commands to Add layers from command line:
source poky/oe-init-build-env
bitbake-layers add-layer ../your_layers

Build an image (check avialable receipes in meta-raspberrypi/recipes-core/images):
bitbake rpi-hwup-image

After finishing building process write image to sdcard of Raspberry:
sudo dd if=./tmp/deploy/images/raspberrypi/rpi-hwup-image-raspberrypi.rpi-sdimg of=/dev/mmcblk0

How to install apt-get and packages in Yocto:
1. In "build/conf/local.conf" add these lines:
PACKAGE_CLASSES = "package_deb"
PACKAGE_FEED_URIS = "http://<ip-of-machine-host-packages>:5678"
EXTRA_IMAGE_FEATURES += " package-management "
2. Build the image (e.g: core-image-sato):
bitbake core-image-sato
3. Create the package index:
bitbake package-index
4. Start packages server so the client can download packages:
cd build/tmp/deploy/deb
python -m SimpleHTTPServer 5678
5. In target side (target that run image), update apt and install:
apt-get update
6. In order to install new packages (e.g: cmake):
- On server side:
bitbake cmake
bitbake package-index
- On target side:
apt-get update
apt-cache search cmake
apt-get install cmake

Clean image when changing recipes:
- Yocto is not designed for development but rather for distribution it won’t strictly follow up on files being changed. You would need to clean and then bake again so make sure that the changes are reflected. Command:
bitbake –c clean <RECIPE OR IMAGE>
bitbake < RECIPE OR IMAGE >

Read More

Parts of Adaptive Autosar

Abbreviation
  - Adaptive Applications (AA)
  - AUTOSAR Runtime for Adaptive applications (ARA)
  - Execution Management (EM)
  - Communication Management (CM)
  - Operating System (OS)
  - Inter-Process-Communication (IPC)
  - Operating System Interface (OSI)

Concepts:
- Language bindings are wrapper libraries that bridge two programming languages, so that a library written for one language can be used in another language
- PSE51 (a single-process profile) is a subset of POSIX created for the purpose of embedded real-time systems. A description can be found in IEEE 1003.13-2003.
- The AUTOSAR adaptive platform is not going to become like Linux, but Linux (or other POSIX-compliant OS) can be taken to host an AUTOSAR adaptive platform.

Adaptive Platform defines an execution context and OSI (contains standard application interface) for use by AAs.

Figure: AP platform

AA run on top ARA
=> ARA consists of application interfaces provided by Functional Clusters
  => Functional Clusters belong to Adaptive Platform Foundation
    => Adaptive Platform Foundation = fundamental functionalities + platform standard services (Adaptive Platform Services)
AA can provide Services (Non-platform service) to other AA
=> AA and Functional Clusters may use non-standard interfaces, but ensure not conflict with the standard AP functionalities and portability onto other AP implementations

ARA implementation ???
  - The language binding is based on C++
  - The C++ Standard library
  - Only PSE51 interface (OS API)

Life-cycles of AA ???
=> Application launch and shutdown
 - Managed by EM (Loading/launching an AA)
  => all the Functional Clusters are applications from EM point of view
 - To launch an AA needs configuration at system integration time or at runtime

Interaction between AAs ???
  - Service requests/replies
    => uses CM for both intra-machine and inter-machine and transparent to applications.

Interaction between Functional Clusters and AAs ???
  - uses IPC under "Library-based" design or "Service-based" design
    + "Library-based" calls IPC directly
      => Uses "Library-based" locally in an AP instance
    + "Service-based" uses CM functionality (AA <=> Cm or AA <=> Server proxy <=> CM)
      => Uses "Service-based" for other AP instance (distributed fashion)

Interaction between Functional Clusters ???
  - use "public" ARA interfaces of other Functional Clusters
  - use "protected" interfaces of Functional Clusters to provide privileged access

AP Operating System point of view ???
=> required to provide multi-process POSIX OS capability
=> the AP and AA are just a set of processes
  => Each AA is implemented as an independent process
  => Functional Clusters are implemented as processes
  => These processes interact with each other through IPC or OS functionalities
  => The Adaptive Platform Services and the non-platform Services are also implemented as processes
    => These processes can have single-threaded or multi-threaded
  => If AAs running on top of ARA, they should only use PSE51.
    => If more features are needed they will be taken from the POSIX standard
  => If a process is the Functional Clusters, it can use available OS API or further POSIX calls
    => The use of specific calls will be left open to the implementer and not standardized.
=> the OS provide interfaces, such as creating processes, that are required by Execution Management to start an Application
  => They are platform implementation dependent (not belong to ARA)
  => OSI provides both C and C++ interfaces.
    => C program includes C function calls defined in the POSIX standard, namely PSE51 defined in IEEE1003.13 [1]
    => C++ program includes function calls defined in the C++ Standard and its Standard C++ Library

Scheduling ???
 - Use standard scheduling policies of POSIX standard : SCHED_FIFO and SCHED_RR
 - Other scheduling policies are allowed, but check portablity across different AP implementations
 
AP runs on as a Machine/hardware ???
=> a hardware can be one or more Machines and a single instance of AP runs on a machine
=> hardware may be virtualized using Hypervisor

Memory management ???
Ref: https://stackoverflow.com/questions/12488010/why-the-entry-point-address-in-my-executable-is-0x8048330-0x330-being-offset-of
  - The process’s address space is virtualized.
  - Each process has its own address space (the addresses where code and data are located may or may not correspond to their underlying physical storage address)

Execution Management ???
  - Working in cooperation with OS
    => responsible for platform initialization and the start-up/shut-down of Applications
    => run-time scheduling of AAs
  - Initialization process : OS first -> Execution Management -> Platform-level Applications -> AAs
  - The startup order of the Platform-level Applications and the AAs are based on Machine Manifest and Application Manifest information
  - Depending on the Machine State, deployed AAs are started during Platform-level Applications startup or later

Figure: AP start-up process

Machine State Management ???
  - Machine State Management define the state of the operation for an Adaptive Platform.
  - The Application Manifest allows definition in which Machine State the Application Executables shall run
  - Grants full control over the set of Applications to be executed
  - Composed of mandatory and non-standardized machine states
  - Machine State Management can be integrated directly in the EM or as a separate Machine State Management Application.

Communication Management ???
=> how a defined service is presented to the application implementer (upper layer, Language Binding) as well as the respective representation of the service’s data on the network (lower layer, Network Binding).
  - Service Oriented Communication (intra-machine communication and inter-machine communication)
    + A service consists of : Events, Methods, Fields
    + Need a Service Registry acts as a brokering instance (services registers, services querying-Service Discovery)

 Figure: Service Oriented Communication

  - Language binding and Network binding
      + Language binding : the methods, events and fields of a service are translated into directly accessible identifiers of the targeted programming language
      + The Network Binding : data of a configured service is serialized and bound to a specific network
      + Service Registry is also part of the Network Binding
      + The interface between Language Binding and Network Binding is private interface inside Communication Management software
=> Generated Proxies and Skeletons of C++ Language Binding ???
  - The upper layer interface of the C++ Language Binding provides an object-oriented mapping of the services
  - A generator that is part of the development tooling for the Communication Management software generates C++ classes
      + On the service implementation side these generated classes are named Service Provider Skeletons.
      + On the client side, these generated classes are called Service Requester Proxies.
  - Service Requester Proxy provides mechanisms for synchronous (blocking the caller until the server returns a result) and asynchronous calling (called function returns immediately).
  - Proxy classes can be used directly by the client
  - The Service Provider Skeletons are abstract classes. A service implementation shall derive from the generated base class and implement the respective functionality.
=> Communication paths configuration ???
  - Only Full service discovery in the application is available in Release 17-03: No communication paths are known at configuration time. An API for Service discovery allows the application code to choose the service instance at runtime.

Figure: Language binding and Network binding

Diagnostics ???
  - The Diagnostic Management realizes the ISO 14229-5 (UDSonIP) which is mainly based on the ISO 14229-1 (UDS) and ISO 13400-2 (DoIP).

Persistency ???
=> Offers a library based approach to access the non-volatile memory for Applications to store data over boot and ignition cycles.
  - Fulfill the PSE51 requirements not to create or delete files during runtime
  - Currently the used storage location identifiers are simply file names
=> 2 categories of storage locations: Key-Value Storage and Stream Storage
  - Key-Value Storage : store and retrieve multiple Key-Value pairs in one storage location.
    + The supported value types are base types, PODs (C++ Plain Old Data structures) and arrays/containers derived from these types, AUTOSAR data types
  - Stream Storage : raw access to file like structures
 
Safety ???
  - protect the exchange of information inside the vehicle and with the external world including fault detection if any corruption has occurred but no mechanisms to guarantee the integrity of data (e.g: E2E-Protection)
  - monitor the correct execution of platform functionalities and AAs
  - safe and secure usage of programming languages
  - Watchdog functionality : Alive supervision, Deadline supervision, Logical supervision,  Error handling of supervision errors

Read More

Adaptive Autosar - Basis points

Abbreviation
  AUTOSAR Adaptive Platform (AP)
  AUTOSAR Classic Platform (CP)
  Service-Oriented-Architecture (SOA)
  Graphics Processing Unit (GPU)
  Field-Programmable Gate Array (FPGA)

Why AP ???
=> Intelligent ECUs
  - AP employs more proven technologies that are not available on traditional ECUs => maximize in the AP implementation to leverage the innovative technologies.
  - There are two major groups of technology drivers:
    + Ethernet
    + Manycore processors And GPU or FPGA  is higher performance than the conventional MCUs

Programming language for development ???
  - C++ (C++14): bring the faster adaptation of novel algorithms and improve application development productivity, IF properly employed.
  - Language bindings AP may support in future
    + Language bindings are wrapper libraries that bridge two programming languages, so that a library written for one language can be used in another language.

Which Architecture ???
  - AP follows service-oriented-architecture (SOA)
  - The SOA concept:
    + A system = services + applications that uses one or more of the services depending on its needs
    System of systems : dedicated systems that pool their resources and capabilities together to create a new, more complex system (more functionality and performance than simply the sum of the constituent systems)
    + A service may reside on local ECU that the application runs, or it can be on a remote ECU, which is also running another instance of AP.
      => The communication infrastructure will take care of the difference so It provides the transparent communication
       => communicating over some form of message passing (fast and high-bandwidth communication such as Ethernet) => USES SOME/IP (service oriented communication over a network)

How Parallel processing ???
=> manycore processors and heterogeneous computing
  - Hardware
  - platform interface specification
  - advancements in OS/hypervisor technologies
  - development tools such as automatic parallelization tools

Make new standard ???
=> focus in developing the AP specification not introduce a new replacement
  - AP takes the strategy of reusing and adapting the existing open standards
    + To facilitate the faster development of the AP
    + Benefiting from the eco-systems of existing standards
 
Safety and security ???
=> possibly at its highest level
 - SOA makes each component more independent and free of unintended interferences
 - C++ coding guideline, which facilitates the safe and secure usage of complex language like C++

Deployment ???
=> incremental deployment => reduce the effort for software development and integration
  - SOA benefits
  - Resources and Communications are managed dynamically

Integration of Classic, Adaptive and Non-AUTOSAR ECUs ???
=> uses SOME/IP for communication among these

AUTOSAR Adaptive Platform - AP
=> AP combines architectural, functional, and procedural approaches.
  + Less modules, only API specification
  + Developed in C++
  + Services as POSIX processes, separately installable
  + Service oriented communication (SOME/IP)
  + Configuration loaded from manifest files

Figure: AUTOSAR Adaptive Platform (AP)

Read More