Havic Kernel and Userspace Migration Plan

Plan to transition the Havic and Gen operating environment

1. Move to Debian 10 or 11 distribution kernel and userspace (64-bit)
   1. Move off of Gentoo based 32-bit environment. The Gentoo distribution is a non-release paradigm distribution, and therefore requires too much local historical knowledge to maintain a stable but non-stale platform without chaining someone to the task of maintenance and updating.
   2. Removes the requirement of creating and maintaining a custom-configured kernel for the purpose of extraordinary longevity since a relatively fresh kernel will be afforded at rational time intervals (every two years +/- 6 months).
2. Utilize standard Debian cross-build packages for ARM for cross-building (will require some work to make the necessary changes to the build procedure), dropping the current “golden tarball” cross build environment kept in the git repository. This means the cross build environment no longer need be created and built utilizing significant engineering resources.
3. Utilize VBox and BTRFS file system as a fallback strategy for pieces that can't be changed to work with the new userspace

Userspace and Kernel periodic upgrade/update plan

1. Develop a suite of tools to automate building, upgrading and updating the software operating environment userspace and kernel, and for installing those upgrades and updates on the Havic and Gen boxes. Motivation: to prevent the software environment of the tester boxes from aging out.
   1. Upgrade to the next major Debian release every two years (approximately). Upgrade will be approached after the first few point updates have occurred for that release to insure that any initial stability problems with that release have been found and addressed.
      1. This upgrade is the result of work to be done by a software engineer. In other words, preparing the upgrade is a hands-on task.
      2. Performed on a specific “primary” build machine/environment and sent out to the Havic and Gen tester boxes. The system will include a “roll-back” feature whereby if the upgrade proves to be too disruptive on an immediate basis per individual box, the environment can be rolled back to the previous version. The roll back operation will require just a few seconds followed by a reboot. The new upgrade environment will still be on the system, allowing for a roll forward at a later time once issues have been resolved. If a box is in a roll-back state, then it will not receive any upgrades or updates until it is on the newest release. The new upgrade environment can be discarded and a new upgrade release installed if that turns out to be necessary.
      3. Custom kernel drivers may have build issues at this time, and software engineering resource(s) applied to address those issues.
      4. Host side custom applications may need software engineering attention at this juncture.
      5. Cross build issues may arise at this time, and hands on software engineering needed to resolve.
      6. Update out-of-distribution-software, like buildroot
   2. Install security and bug fix updates every six months or sooner as needed. Bug fixes include only the most dire lack of functionality type bugs. Custom host side kernel drivers will be automatically rebuilt and installed with each update of the kernel. These updates are considered minor and should not bring about any issues. Kernel updates should not include changes to the kernel APIs used by the custom host side drivers unless those changes are considered critical for proper operation of the system or to prevent data loss.
      1. Performed by user on individual tester boxes. If desired, the build machine tools can include a protocol to contact the individual tester boxes and signal them to perform this update, however this may not always be possible as those machines may not be able to communicate with the primary build machine. Such would not be an automated function. It would be triggered by a user on the primary build machine.
      2. These updates will not be part of the roll-back/roll-forward functionality.
2. Thoroughly document the design and methodology of the environment, and synopsis of the tools.