Connect to the serial console

The FTDI bridge on board provides a USB serial console on the USB-C port.

Settings are the typical 115200,8,N,1 with no flow control.

 minicom -D /dev/ttyUSB

Firmware update

Bootup and enter a key to stop the regular boot process. You are then in the Management Complex:

=> bootmenu

  *** U-Boot Boot Menu ***

     Normal boot
     Built-in recovery
     Boot from NVMe
     Boot from USB
     Boot from SD
     Boot OpenWrt from NAND
     Boot from network
     Boot from SATA
     U-Boot console

  Hit any key to stop autoboot:  6
  Press UP/DOWN to move, ENTER to select

Does allow to select the Built-in recovery.

A complete outdated recovery image started:

BusyBox v1.31.1 () built-in shell (ash)
 __ ___ ___ ___ __ __ ___ _ __ _ _
 | '_*|/ _ \ / _*|/ _ \\ \ / // _ \| '_*|| | | |
 | | | _*/| (_*| (_) |\ V /| _*/| | | |_| |
 |_| \_**| \_**|\_**/ \_/ \_**||_| \_*, |
                                           _*/ |
                                          |___/
 -----------------------------------------------------
 ten64-recovery v0.8.7, 332215009_2021-07-06
 -----------------------------------------------------

Firmware update

Download the latest firmware and copy it to an USB stick with an ext4 formated filesystem.

To flash the firmware within the recovery environment:

mount /dev/sda1 /mnt
cd /mnt/firmware
./flash.sh

Recovery firmware update

The recovery firmware can be updated directly from within the recovery environment.

Download the latest firmware and copy it to an USB stick with an ext4 formated filesystem.

mount /dev/sda1 /mnt
cd /mnt/
mtd erase recovery && mtd write recovery.itb recovery

The original recovery firmware updated by the firmware update:

BusyBox v1.33.1 (2021-10-13 05:51:35 UTC) built-in shell (ash)

_ __ ___ ___ ___ __ __ ___ _ __ _ _
 | '_*|/ _ \ / _*|/ _ \\ \ / // _ \| '_*|| | | |
 | | | _*/| (_*| (_) |\ V /| _*/| | | |_| |
 |_| \_**| \_**|\_**/ \_/ \_**||_| \_*, |
                                           _*/ |
                                          |___/
 -----------------------------------------------------
 ten64-recovery master, 399304623_2021-11-01
 -----------------------------------------------------

The recovery firmware messages after the firmware update:

BusyBox v1.36.1 (2024-02-18 14:57:42 UTC) built-in shell (ash)

_ __ ___ ___ ___ __ __ ___ _ __ _ _
 | '_*|/ _ \ / _*|/ _ \\ \ / // _ \| '_*|| | | |
 | | | _*/| (_*| (_) |\ V /| _*/| | | |_| |
 |_| \_**| \_**|\_**/ \_/ \_**||_| \_*, |
                                           _*/ |
                                          |___/
 -----------------------------------------------------
 ten64-recovery master, 1196480059_2024-03-01
 -----------------------------------------------------

Debian setup

Debian can be installed in various ways. However, the most intelligent approach is to use baremetal-deploy, as it not only installs the image but also applies patches for hardware compatibility issues.

For instance, using the Debian image:

{
    "id" : "debian-stable",
    "name" : "Debian stable (12)",
    "description" : "Debian stable (codename bookworm)",
    "maintainer" : "Debian",
    "download" : "https://cloud.debian.org/images/cloud/bookworm/daily/20250207-
2016/debian-12-generic-arm64-daily-20250207-2016.qcow2",
    "link" : "https://cloud.debian.org/images/cloud/buster/",
    "checksum" : "sha512:8fbe4d093d88a9c621f869622814c01d57553e02a66c4f05f53373f
e8dc86924564e94f77b009c68810d771149bf7a3b4c8a86f790ef595de1983b73defc9fb2",
    "format" : "qcow2",
    "version" : "20250207-2016",
    "mindisk" : 4096,
    "minram" : 1024,
    "supported_hardware" : null,
    "incompat_hardware" : \[ "traverse,ten64" \],
    "cloudinit" : true,
    "properties" : {
      "x-dpaa2-legacy-ethernet" : "false",
      "x-dpaa2-legacy-sfp" : "true"
    }
  }

Configure the WAN uplink:

root@recovery000afa242d1d:/# set-wan eth0
Interface eth0 is a physical port for bridge br-lan.
Remove interface from bridge? [yN] y
Removing eth0 from br-lan
[   50.490448] device eth0 left promiscuous mode
[   50.499504] br-lan: port 1(eth0) entered disabled state

Launch the deployment tool:

root@recovery000afa242d1d:/# baremetal-deploy debian-stable /dev/nvme0n1
Checking compatibility...
NOTE: A manipulator will be used to fix compatibility issues between this image and hardware platform
ID: ten64-apt-manipulator by Traverse Technologies
URL: https://archive.traverse.com.au/pub/arm-image-registry/manipulators/apt/apt-manipulator.lua

Appliance to deploy: debian-stable - Debian stable (12)
Device to deploy to: /dev/nvme0n1 (1863.02GiB)
Downloading image
Image URL: https://cloud.debian.org/images/cloud/bookworm/daily/20250207-2016/debian-12-generic-arm64-daily-20250207-2016.qcow2
Size of the image download: 410M

! ! ! ! ! ! ! ! ! !
About to write image to device /dev/nvme0n1 if you don't want this, Ctrl-C now!
! ! ! ! ! ! ! ! ! !
[  229.586451] nbd0: detected capacity change from 0 to 6291456
[  229.594595]  nbd0: p1 p15
[  229.604058]  nbd0: p1 p15
[  247.194479] block nbd0: NBD_DISCONNECT
[  247.198251] block nbd0: Disconnected due to user request.
[  247.203657] block nbd0: shutting down sockets
What should the user password be? password
Force user to change password on first login (y/n)? [n] y
Change system locale (e.g en_AU, de_DE)? [leave blank to keep default image locale] en_US
What should the hostname be? [leave blank to use appliance default] n-a
Set the default upstream interface? eth0
Use DHCP6 to get an IPv6 address? On a home network that uses router advertisments for IPv6, answer "n" (y/n) [n]y
Setting default user password: gabriel
Hostname: n-c
netintf: eth0
Invoking run-manipulate --userpw=gabriel --hostname=n-c --netintf=eth0 --locale=en_US --dhcpv6 --force-pw-change /tmp/manipulator-script.lua /dev/nvme0n1


------------------------------------------------------
Using manipulator script: /tmp/manipulator-script.lua
Resizing rootfs and adding swap partition
Root partition type is ext4
Setting up target chroot
[  314.803550] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
Adding swap to fstab
Distribution is debian
Setting default kernel cmdline
Executing: chroot /mnt/deployroot/ /bin/sh -l -c locale-gen
Running update-grub
Executing: chroot /mnt/deployroot/ /bin/sh -l -c update-grub
[  330.047755] F2FS-fs (nvme0n1p15): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[  330.055072] F2FS-fs (nvme0n1p15): Can't find valid F2FS filesystem in 1th superblock
[  330.062956] F2FS-fs (nvme0n1p15): Magic Mismatch, valid(0xf2f52010) - read(0x6020601)
[  330.070791] F2FS-fs (nvme0n1p15): Can't find valid F2FS filesystem in 2th superblock
Have network interface: eth0
#cloud-config
password: $6$6FJ7EliYQ3BlGhQR$8geG.xH8yafC3ZyyXEN3oh756B.pQ1wjf/VPmGnnK655y0AKwwHQkLc7ObT8UwA204QyWUfWmVAbuAJF4B3FP/
fqdn: n-c
locale: en_US

Done!
------------------------------------------------------
Manipulator finished
Ensuring managed mode is set for Gigabit Ethernet
Setting legacy management mode for SFP+

Partitioning

We are going to create four partitons:

#	Partition content	Size
1	EFI	127M
2	Linux root	200 GB
3	Ceph OSD	1.5 TB
4	TopoLVM	200 GB

Unfortuantly the partition table is more or less cra**** to use for a Kubernetes node:

root@n-a:~# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme0n1      259:0    0  1.9T  0 disk
|-nvme0n1p1  259:1    0  1.9T  0 part /
|-nvme0n1p2  259:2    0    4G  0 part [SWAP]
`-nvme0n1p15 259:3    0  127M  0 part /boot/efi

Shrink the filesystem:

resize2fs /dev/nvme0n1p1 50G
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/nvme0n1p1 to 13107200 (4k) blocks.
The filesystem on /dev/nvme0n1p1 is now 13107200 (4k) blocks long.

Verify the current partition table:

# parted /dev/nvme0n1 print
Model: ADATA SX8200PNP (nvme)
Disk /dev/nvme0n1: 2048GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
15      1049kB  134MB   133MB   fat16                 boot, esp
 1      134MB   2044GB  2044GB  ext4
 2      2044GB  2048GB  4295MB  linux-swap(v1)  swap

Delete the SWAP patition:

parted --script /dev/nvme0n1 "rm 2"

Check the current partition table with free:

# parted /dev/nvme0n1 print free
Model: ADATA SX8200PNP (nvme)
Disk /dev/nvme0n1: 2048GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
        17.4kB  1049kB  1031kB  Free Space
15      1049kB  134MB   133MB   fat16              boot, esp
 1      134MB   200GB   200GB   ext4
        200GB   200GB   143kB   Free Space
 2      200GB   2048GB  1848GB                     lvm
        2048GB  2048GB  335kB   Free Space

Shrink the partition table:

# parted /dev/nvme0n1

(parted) resizepart 1 200G
Warning: Shrinking a partition can cause data loss, are you sure you want to
continue?
Yes/No? Yes

Reboot the system to absolutly ensure the partiton table has been written and is consistent:

# reboot

Grow the filesystem:

# resize2fs /dev/nvme0n1p1
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/nvme0n1p1 to 48795392 (4k) blocks.
The filesystem on /dev/nvme0n1p1 is now 48795392 (4k) blocks long.

Create the partitoon for Rook and TopoLVM:

parted --script /dev/nvme0n1 "mkpart '' 200GB 1848GB"
parted --script /dev/nvme0n1 "mkpart '' 1848GB 100%"

Set the LVM flog for the TopoLVM partiton:

parted --script /dev/nvme0n1 "set 3 lvm on"

Check the alignment:

# parted /dev/nvme0n1 align-check opt 1
1 aligned

# parted /dev/nvme0n1 align-check opt 2
1 aligned

# parted /dev/nvme0n1 align-check opt 3
1 aligned

Verify the partiton UUIDs and delete the SWAP partition entry

Get the PARTUUID:

# blkid /dev/nvme0n1p1
/dev/nvme0n1p1: UUID="88506ba0-aa46-4972-82fe-e1c27e0ac953" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="c9265917-e0f0-4d10-8694-cc44b47651f6"

Verify if the PARTUUID matches in /etc/fstab:

# mount /dev/nvme0n1p1 /mnt
# cat /mnt/etc/fstab
PARTUUID=c9265917-e0f0-4d10-8694-cc44b47651f6 / ext4 rw,discard,errors=remount-ro,x-systemd.growfs 0 1

Remove the SWAP partiton entry.

Disable cloud-init

On each startup, cloud-init is executed. This can lead to unintended side effects that we need to avoid.

# touch /etc/cloud/cloud-init.disabled

Reduce permissions on netplan from cloud-init

# chmod 600 /etc/netplan/50-cloud-init.yaml

Configure the network

Generate an unique IPv6 subnet.

Add the configuration to the netplan config:

# vim /etc/netplan/50-cloud-init.yaml
network:
    ethernets:
        eth0:
            dhcp4: true
            dhcp6: true
            addresses:
              - fd25:c06e:f337::a/64
    version: 2

Add the SSH key

Of the debian user:

$ echo "ssh-*" >> .ssh/authorized_keys

Performance 10 Gbit network

The DPAA2 network interfaces do support a MTU of up to 10222. Direct connection using a Twinax cabel.

Using an MTU of 1500:

root@n-a:~# iperf3 -c fd5b:107:24a8:ab::b -p 5201
Connecting to host fd5b:107:24a8:ab::b, port 5201
[  5] local fd5b:107:24a8:ab::a port 36996 connected to fd5b:107:24a8:ab::b port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec   115 MBytes   954 Mbits/sec    0   1.20 MBytes
[  5]   1.01-2.01   sec   125 MBytes  1.05 Gbits/sec    0   1.20 MBytes
[  5]   2.01-3.00   sec   125 MBytes  1.06 Gbits/sec    0   1.20 MBytes
[  5]   3.00-4.01   sec   126 MBytes  1.05 Gbits/sec    0   1.20 MBytes
[  5]   4.01-5.00   sec   124 MBytes  1.05 Gbits/sec    0   1.46 MBytes
[  5]   5.00-6.00   sec   125 MBytes  1.05 Gbits/sec    0   1.69 MBytes
[  5]   6.00-7.01   sec   125 MBytes  1.04 Gbits/sec    0   1.69 MBytes
[  5]   7.01-8.01   sec   125 MBytes  1.05 Gbits/sec    0   1.69 MBytes
[  5]   8.01-9.01   sec   125 MBytes  1.05 Gbits/sec    0   1.93 MBytes
[  5]   9.01-10.01  sec   125 MBytes  1.05 Gbits/sec    0   1.93 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.21 GBytes  1.04 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  1.21 GBytes  1.04 Gbits/sec                  receiver

Using an MTU of 9000:

root@n-a:~# iperf3 -c fd5b:107:24a8:ab::b -p 5201
Connecting to host fd5b:107:24a8:ab::b, port 5201
[  5] local fd5b:107:24a8:ab::a port 36334 connected to fd5b:107:24a8:ab::b port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   310 MBytes  2.59 Gbits/sec    0   3.18 MBytes
[  5]   1.00-2.00   sec   345 MBytes  2.90 Gbits/sec    0   3.18 MBytes
[  5]   2.00-3.00   sec   354 MBytes  2.97 Gbits/sec    0   3.18 MBytes
[  5]   3.00-4.00   sec   355 MBytes  2.97 Gbits/sec    0   3.18 MBytes
[  5]   4.00-5.00   sec   348 MBytes  2.92 Gbits/sec    0   3.18 MBytes
[  5]   5.00-6.00   sec   356 MBytes  2.99 Gbits/sec    0   3.18 MBytes
[  5]   6.00-7.00   sec   351 MBytes  2.95 Gbits/sec    0   3.18 MBytes
[  5]   7.00-8.00   sec   348 MBytes  2.92 Gbits/sec    0   3.18 MBytes
[  5]   8.00-9.00   sec   349 MBytes  2.93 Gbits/sec    0   3.18 MBytes
[  5]   9.00-10.00  sec   352 MBytes  2.95 Gbits/sec    0   3.18 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.39 GBytes  2.91 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  3.38 GBytes  2.91 Gbits/sec                  receiver

Using an MTU of 10222:

root@n-a:~# iperf3 -c fd5b:107:24a8:ab::b -p 5201
Connecting to host fd5b:107:24a8:ab::b, port 5201
[  5] local fd5b:107:24a8:ab::a port 42244 connected to fd5b:107:24a8:ab::b port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   305 MBytes  2.55 Gbits/sec    0   3.13 MBytes
[  5]   1.00-2.00   sec   350 MBytes  2.94 Gbits/sec    0   3.13 MBytes
[  5]   2.00-3.00   sec   359 MBytes  3.02 Gbits/sec    0   3.13 MBytes
[  5]   3.00-4.00   sec   358 MBytes  3.00 Gbits/sec    0   3.13 MBytes
[  5]   4.00-5.00   sec   376 MBytes  3.15 Gbits/sec    0   3.13 MBytes
[  5]   5.00-6.00   sec   375 MBytes  3.15 Gbits/sec    0   3.13 MBytes
[  5]   6.00-7.00   sec   358 MBytes  3.00 Gbits/sec    0   3.13 MBytes
[  5]   7.00-8.00   sec   361 MBytes  3.03 Gbits/sec    0   3.13 MBytes
[  5]   8.00-9.00   sec   356 MBytes  2.98 Gbits/sec    0   3.13 MBytes
[  5]   9.00-10.00  sec   352 MBytes  2.96 Gbits/sec    0   3.13 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.47 GBytes  2.98 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  3.46 GBytes  2.98 Gbits/sec                  receiver

Ten64 - Firmware update and Debian setup

Connect to the serial console

Firmware update

Firmware update

Recovery firmware update

Debian setup

Partitioning

Verify the partiton UUIDs and delete the SWAP partition entry

Disable cloud-init

Reduce permissions on netplan from cloud-init

Configure the network

Add the SSH key

Performance 10 Gbit network