Customize /system mountpoint on Android

Question

I'm trying to change a Nexus 5 to use dm-verity module. For that I need to:

1. invoke an utility called veritysetup during boot before /system is mounted;
2. (this utility will will create a new device called /dev/mapper/devname);
3. let /system be mounted on /dev/mapper/devname, not the actual partition.

It looks like the mountpoint is configured on fstab.hammerhead (device/lge/hammerhead). In fact I have another mod working by changing this file. So this seams like the easy part.

I know some filesystems are mounted by init.c (system/core/init), like /dev and /proc, and some are mounted on init.hammerhead (system/core/rootdir), like /acct. But I can find where the request to mount /system is.

Any ideas?

Answer 1

That's not the correct way to do it. To use dm-verity we need to:

1. build android using "user" build type (default is "eng");
2. change fstab to include "verify" keyword on fs_mgr_flags.

So, when Android the vold module will find the verity flag and will create the mapper device.

If the system image has a signed hash table and precomputed hash and the root image has the public key everything should just work. We should find that the system partition was mounted on /dev/block/dm-0 instead of /dev/block/platform/msm_sdcc.1/by-name/system and is being verified while being read.

Answer 2

I am having a similar inquiry and the posted answer here is closest related post I have found, certaintly my inquiry is relavant...

Does anyone know where the code that generates or places the fstab files into the stage1 and stage2 boots is, I think there is a bug or missing file somewhere. I can easily change and update things. What file in AOSP actually needs to be edited before building in order to modify or make changes in the fstab file before it is packed into he ramfs stage 1 or stage 2. Also where do I place kernel debug parameters like "pci=nocrs" ?

*Note: The answer accepted here is the correct answer per the aosp source documentation, but it leaves out the same valuable information the source documentation does. Where does the file need to be placed in the tree before running the build I can see the output does have vendor folder with a couple versions of fstab, but I am not sure where it was placed in the boot images.

I would like to boot this in qemu kvm with the cuttlefish build. Full kernel.log file here: https://pastebin.com/9MKFKeyN Intresting part below:

[    1.702829] init: Switching root to '/first_stage_ramdisk'
[    1.703038] init: [libfs_mgr]ReadFstabFromDt(): failed to read fstab from dt
[    1.703574] init: Using Android DT directory /proc/device-tree/firmware/android/
[    1.712562] init: [libfs_mgr]Invalid ext4 superblock on '/dev/block/by-name/metadata'
[    1.712864] traps: init[1] trap invalid opcode ip:41a6eb sp:7ffff59c93e0 error:0 in init[2f7000+200000]
[    1.713197] init: InitFatalReboot: signal 4
[    1.713574] init: #00 pc 000000000031b86b  /init
[    1.713626] init: Reboot ending, jumping to kernel
[    1.713718] kvm: exiting hardware virtualization
[    1.788726] reboot: Restarting system with command 'bootloader'
[    1.788872] reboot: machine restart

Latest AOSP build as of 4/2/2022

buildprops aosp/out/target/product/vsoc_x86_64/ramdisk/system/etc/ramdisk/build.prop

####################################
# from generate-common-build-props
# These properties identify this partition image.
####################################
ro.product.bootimage.brand=generic
ro.product.bootimage.device=vsoc_x86_64
ro.product.bootimage.manufacturer=Google
ro.product.bootimage.model=Cuttlefish x86_64 tv
ro.product.bootimage.name=aosp_cf_x86_64_tv
ro.bootimage.build.date=Sat Apr  2 12:06:36 CDT 2022
ro.bootimage.build.date.utc=1648919196
ro.bootimage.build.fingerprint=generic/aosp_cf_x86_64_tv/vsoc_x86_64:Tiramisu/AOSP.MASTER/me04021206:userdebug/test-keys
ro.bootimage.build.id=AOSP.MASTER
ro.bootimage.build.tags=test-keys
ro.bootimage.build.type=userdebug
ro.bootimage.build.version.incremental=eng.me.20220402.120943
ro.bootimage.build.version.release=12
ro.bootimage.build.version.release_or_codename=Tiramisu
ro.bootimage.build.version.sdk=32
# end of file

fstab.ext4 /genvol/aosp/out/target/product/vsoc_x86_64/recovery/root/first_stage_ramdisk/fstab.ext4

# Non-dynamic, boot critical partitions
/dev/block/by-name/boot /boot emmc defaults recoveryonly,slotselect,first_stage_mount,avb=boot
/dev/block/by-name/init_boot /init_boot emmc defaults recoveryonly,slotselect,first_stage_mount,avb=init_boot
/dev/block/by-name/vendor_boot /vendor_boot emmc defaults recoveryonly,slotselect
system /system erofs ro wait,logical,first_stage_mount,slotselect,avb=vbmeta_system,avb_keys=/avb
system /system ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb=vbmeta_system,avb_keys=/avb
# Add all non-dynamic partitions except system, after this comment
/dev/block/by-name/userdata /data ext4 nodev,noatime,nosuid,errors=panic latemount,wait,check,quota,formattable,fileencryption=aes-256-xts:aes-256-cts,keydirectory=/metadata/vold/metadata_encryption,checkpoint=block
/dev/block/by-name/metadata /metadata ext4 nodev,noatime,nosuid,errors=panic wait,formattable,first_stage_mount,check
/dev/block/by-name/misc /misc emmc defaults defaults
# Add all dynamic partitions except system, after this comment
odm /odm erofs ro wait,logical,first_stage_mount,slotselect,avb
odm /odm ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb
product /product erofs ro wait,logical,first_stage_mount,slotselect,avb
product /product ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb
system_ext /system_ext erofs ro wait,logical,first_stage_mount,slotselect,avb=vbmeta_system
system_ext /system_ext ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb=vbmeta_system
vendor /vendor erofs ro wait,logical,first_stage_mount,slotselect,avb=vbmeta
vendor /vendor ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb=vbmeta
vendor_dlkm /vendor_dlkm erofs ro wait,logical,first_stage_mount,slotselect,avb
vendor_dlkm /vendor_dlkm ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb
odm_dlkm /odm_dlkm erofs ro wait,logical,first_stage_mount,slotselect,avb
odm_dlkm /odm_dlkm ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb
system_dlkm /system_dlkm erofs ro wait,logical,first_stage_mount,slotselect,avb=vbmeta
system_dlkm /system_dlkm ext4 noatime,ro,errors=panic wait,logical,first_stage_mount,slotselect,avb=vbmeta
# ZRAM, SD-Card and virtiofs shares
/dev/block/zram0 none swap defaults zramsize=75%
/dev/block/vdc1 /sdcard vfat defaults recoveryonly
/devices/*/block/vdc auto auto defaults voldmanaged=sdcard1:auto,encryptable=userdata
shared /mnt/vendor/shared virtiofs nosuid,nodev,noatime nofail

I also have full kernel log and other information if it will help, Vtd and Vtx are working and all of the pci show in the IOMMU Groups.

acloud create --local-instance 1 --local-image tells me how to start troubleshooting when it fails, only after you manually build the certfile as that part of the acloud setup for instance1 cannot complete, I found that error, but have not yet fixed the source for it, that one was simple - Manually created the cert allowed it to proceed but fails at the switchroot in the boot. All the hardware and qemu all showed good.

A few points of interest as cannot add full kernel.log

Waiting for AVD(s) to boot up ...stop_cvd I 04-02 17:33:50 137759 137759 main.cc:162] Successfully stopped device cvd-1: 0.0.0.0:6520
Fail! (453s)
Total time:  (453s)


Device summary:
Fail in:
Cannot create cuttlefish instance: Device did not boot within 450 secs. Stderr:
The following files contain useful debugging information:
  Serial console is disabled; use -console=true to enable it.
  Kernel log: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/kernel.log
  Logcat output: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/logs/logcat
  Launcher log: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/logs/launcher.log
  Instance configuration: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/cuttlefish_config.json
  Instance environment: /tmp/acloud_cvd_temp/local-instance-1/.cuttlefish.sh
Failed to read a complete exit code, read 0 bytes only instead of the expected 4
VIRTUAL_DEVICE_BOOT_FAILED
launch_cvd E 04-02 17:33:50 115944 115944 main.cc:252] run_cvd returned 10
For more detail: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/launcher.log

Encountered the following errors:
Cannot create cuttlefish instance: Device did not boot within 450 secs. Stderr:
The following files contain useful debugging information:
  Serial console is disabled; use -console=true to enable it.
  Kernel log: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/kernel.log
  Logcat output: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/logs/logcat
  Launcher log: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/logs/launcher.log
  Instance configuration: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/instances/cvd-1/cuttlefish_config.json
  Instance environment: /tmp/acloud_cvd_temp/local-instance-1/.cuttlefish.sh
Failed to read a complete exit code, read 0 bytes only instead of the expected 4
VIRTUAL_DEVICE_BOOT_FAILED
launch_cvd E 04-02 17:33:50 115944 115944 main.cc:252] run_cvd returned 10
For more detail: /tmp/acloud_cvd_temp/local-instance-1/cuttlefish_runtime/launcher.log

If you have any question or need acloud team support, please feel free to contact us by email at [email protected].


Line 227 of kernel log: [    0.318334] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug