Initramfs investigation
Investigate initramfs issues - extraction, module analysis, and comparing working vs failing buildsFrom its SKILL.md
npx -y skills add coreos/ai-helpers --skill initramfs-investigationAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 7 stars7 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
5.9 KB, as published. Nobody here has run it
Initramfs Investigation
Knowledge for investigating initramfs-related boot failures by comparing working and failing builds in FCOS and RHCOS.
Related:
pipeline-failures(CI failures),rhcos-cosa(cosa builds),rhcos-artifacts(build artifacts)
Initramfs Extraction
Identify Compression Format
# Check magic bytes
head -c 6 initramfs.img | od -A x -t x1z
| Magic Bytes | Format | Tool |
|---|---|---|
28 b5 2f fd | zstd | zstdcat |
1f 8b | gzip | zcat |
fd 37 7a 58 | xz | xzcat |
30 37 30 37 | CPIO (uncompressed) | Direct cpio |
Extract Initramfs
# For zstd-compressed (FCOS/RHCOS default)
mkdir extracted && cd extracted
zstdcat ../initramfs.img | cpio -idm
# For gzip-compressed
zcat ../initramfs.img | cpio -idm
# For multi-segment (early microcode + main)
skipcpio initramfs.img | zstdcat | cpio -idm
List Contents Without Extraction
# Using lsinitrd
lsinitrd initramfs.img
# Manual listing
zstdcat initramfs.img | cpio -t | head -50
Kernel Module Analysis
Find Modules in Initramfs
# List kernel modules
find extracted -name "*.ko*" | head -20
# Find specific module
find extracted -name "scsi_transport_iscsi*"
Examine Module
# Decompress module first (if .ko.xz)
xz -d extracted/usr/lib/modules/*/kernel/drivers/scsi/module.ko.xz
# List all ELF sections
readelf -S module.ko
# Check module architecture
readelf -h module.ko | grep -E "Data|Machine|Class"
FCOS Build Artifacts
Download Initramfs
# List available builds
curl -s "https://builds.coreos.fedoraproject.org/prod/streams/<stream>/builds/builds.json" | jq -r '.builds[].id' | head -10
# Get build metadata
curl -s "https://builds.coreos.fedoraproject.org/prod/streams/<stream>/builds/<version>/<arch>/meta.json" | jq .
# Download initramfs
curl -LO "https://builds.coreos.fedoraproject.org/prod/streams/<stream>/builds/<version>/<arch>/fedora-coreos-<version>-live-initramfs.<arch>.img"
Streams: stable, testing, next, rawhide
Compare Package Versions Between Builds
# Get package diff from meta.json (shows what changed from previous build)
curl -s ".../meta.json" | jq '.pkgdiff'
# Check specific package (e.g., kernel)
curl -s ".../meta.json" | jq '.pkgdiff[] | select(.[0] | test("kernel"))'
RHCOS Build Artifacts
From RHCOS Release Browser (Internal)
Base URL: https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com
# List available builds for a stream
curl -s "https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/storage/prod/streams/<stream>/builds/builds.json" | jq -r '.builds[].id' | head -10
# Get build metadata
curl -s "https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/storage/prod/streams/<stream>/builds/<buildid>/<arch>/meta.json" | jq .
# Get initramfs path from metadata
curl -s ".../meta.json" | jq -r '.images["live-initramfs"].path'
# Download initramfs
curl -LO "https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/storage/prod/streams/<stream>/builds/<buildid>/<arch>/<initramfs-filename>"
Streams: rhel-9.4, rhel-9.6, rhel-10.0, rhel-10.2, etc.
Comparing Working vs Failing Initramfs
1. Identify Working and Failing Builds
From CI failure logs or build history, identify:
- Last working build (e.g.,
45.20260323.91.1) - First failing build (e.g.,
45.20260324.91.1)
2. Download Both Initramfs
# FCOS example
curl -LO ".../45.20260323.91.1/<arch>/fedora-coreos-45.20260323.91.1-live-initramfs.<arch>.img"
curl -LO ".../45.20260324.91.1/<arch>/fedora-coreos-45.20260324.91.1-live-initramfs.<arch>.img"
mv *323* initramfs-working.img
mv *324* initramfs-failing.img
3. Extract Both
mkdir working failing
cd working && zstdcat ../initramfs-working.img | cpio -idm 2>/dev/null && cd ..
cd failing && zstdcat ../initramfs-failing.img | cpio -idm 2>/dev/null && cd ..
4. Compare Kernel Versions
ls working/usr/lib/modules/
ls failing/usr/lib/modules/
5. Compare Specific Files
# Compare dracut modules included
diff <(ls working/usr/lib/dracut/modules.d/) <(ls failing/usr/lib/dracut/modules.d/)
# Compare a specific module's ELF sections
diff <(readelf -S working/usr/lib/modules/*/kernel/path/to/module.ko) \
<(readelf -S failing/usr/lib/modules/*/kernel/path/to/module.ko)
# Compare file sizes
find working -type f -name "*.ko*" -exec ls -la {} \; | sort > /tmp/working-modules.txt
find failing -type f -name "*.ko*" -exec ls -la {} \; | sort > /tmp/failing-modules.txt
diff /tmp/working-modules.txt /tmp/failing-modules.txt
6. Check dracut Configuration
# Compare dracut config
diff working/usr/lib/dracut/dracut.conf.d/ failing/usr/lib/dracut/dracut.conf.d/
# Check specific dracut scripts
diff working/usr/lib/dracut/modules.d/95iscsi/parse-iscsiroot.sh \
failing/usr/lib/dracut/modules.d/95iscsi/parse-iscsiroot.sh
Analyzing Console Logs
When investigating boot failures from kola tests:
# Extract kola artifacts
tar -xf kola-upgrade-<arch>.tar.xz
# Find console logs
find . -name "console.txt"
# Look for first error (not cascade failures)
grep -n -E "FATAL|failed to|Failed to|error:" console.txt | head -10
# Check module loading
grep -E "modprobe|insmod|systemd-modules-load" console.txt