agentsclimarketplace

Initramfs investigation

Skill coreos/ai-helpers/skills/initramfs-investigation

Investigate initramfs issues - extraction, module analysis, and comparing working vs failing buildsFrom its SKILL.md

Install
npx -y skills add coreos/ai-helpers --skill initramfs-investigation

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 7 stars7 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

5.9 KB, as published. Nobody here has run it

Initramfs Investigation

Knowledge for investigating initramfs-related boot failures by comparing working and failing builds in FCOS and RHCOS.

Related: pipeline-failures (CI failures), rhcos-cosa (cosa builds), rhcos-artifacts (build artifacts)

Initramfs Extraction

Identify Compression Format

# Check magic bytes
head -c 6 initramfs.img | od -A x -t x1z
Magic BytesFormatTool
28 b5 2f fdzstdzstdcat
1f 8bgzipzcat
fd 37 7a 58xzxzcat
30 37 30 37CPIO (uncompressed)Direct cpio

Extract Initramfs

# For zstd-compressed (FCOS/RHCOS default)
mkdir extracted && cd extracted
zstdcat ../initramfs.img | cpio -idm

# For gzip-compressed
zcat ../initramfs.img | cpio -idm

# For multi-segment (early microcode + main)
skipcpio initramfs.img | zstdcat | cpio -idm

List Contents Without Extraction

# Using lsinitrd
lsinitrd initramfs.img

# Manual listing
zstdcat initramfs.img | cpio -t | head -50

Kernel Module Analysis

Find Modules in Initramfs

# List kernel modules
find extracted -name "*.ko*" | head -20

# Find specific module
find extracted -name "scsi_transport_iscsi*"

Examine Module

# Decompress module first (if .ko.xz)
xz -d extracted/usr/lib/modules/*/kernel/drivers/scsi/module.ko.xz

# List all ELF sections
readelf -S module.ko

# Check module architecture
readelf -h module.ko | grep -E "Data|Machine|Class"

FCOS Build Artifacts

Download Initramfs

# List available builds
curl -s "https://builds.coreos.fedoraproject.org/prod/streams/<stream>/builds/builds.json" | jq -r '.builds[].id' | head -10

# Get build metadata
curl -s "https://builds.coreos.fedoraproject.org/prod/streams/<stream>/builds/<version>/<arch>/meta.json" | jq .

# Download initramfs
curl -LO "https://builds.coreos.fedoraproject.org/prod/streams/<stream>/builds/<version>/<arch>/fedora-coreos-<version>-live-initramfs.<arch>.img"

Streams: stable, testing, next, rawhide

Compare Package Versions Between Builds

# Get package diff from meta.json (shows what changed from previous build)
curl -s ".../meta.json" | jq '.pkgdiff'

# Check specific package (e.g., kernel)
curl -s ".../meta.json" | jq '.pkgdiff[] | select(.[0] | test("kernel"))'

RHCOS Build Artifacts

From RHCOS Release Browser (Internal)

Base URL: https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com

# List available builds for a stream
curl -s "https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/storage/prod/streams/<stream>/builds/builds.json" | jq -r '.builds[].id' | head -10

# Get build metadata
curl -s "https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/storage/prod/streams/<stream>/builds/<buildid>/<arch>/meta.json" | jq .

# Get initramfs path from metadata
curl -s ".../meta.json" | jq -r '.images["live-initramfs"].path'

# Download initramfs
curl -LO "https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/storage/prod/streams/<stream>/builds/<buildid>/<arch>/<initramfs-filename>"

Streams: rhel-9.4, rhel-9.6, rhel-10.0, rhel-10.2, etc.

Comparing Working vs Failing Initramfs

1. Identify Working and Failing Builds

From CI failure logs or build history, identify:

  • Last working build (e.g., 45.20260323.91.1)
  • First failing build (e.g., 45.20260324.91.1)

2. Download Both Initramfs

# FCOS example
curl -LO ".../45.20260323.91.1/<arch>/fedora-coreos-45.20260323.91.1-live-initramfs.<arch>.img"
curl -LO ".../45.20260324.91.1/<arch>/fedora-coreos-45.20260324.91.1-live-initramfs.<arch>.img"

mv *323* initramfs-working.img
mv *324* initramfs-failing.img

3. Extract Both

mkdir working failing

cd working && zstdcat ../initramfs-working.img | cpio -idm 2>/dev/null && cd ..
cd failing && zstdcat ../initramfs-failing.img | cpio -idm 2>/dev/null && cd ..

4. Compare Kernel Versions

ls working/usr/lib/modules/
ls failing/usr/lib/modules/

5. Compare Specific Files

# Compare dracut modules included
diff <(ls working/usr/lib/dracut/modules.d/) <(ls failing/usr/lib/dracut/modules.d/)

# Compare a specific module's ELF sections
diff <(readelf -S working/usr/lib/modules/*/kernel/path/to/module.ko) \
     <(readelf -S failing/usr/lib/modules/*/kernel/path/to/module.ko)

# Compare file sizes
find working -type f -name "*.ko*" -exec ls -la {} \; | sort > /tmp/working-modules.txt
find failing -type f -name "*.ko*" -exec ls -la {} \; | sort > /tmp/failing-modules.txt
diff /tmp/working-modules.txt /tmp/failing-modules.txt

6. Check dracut Configuration

# Compare dracut config
diff working/usr/lib/dracut/dracut.conf.d/ failing/usr/lib/dracut/dracut.conf.d/

# Check specific dracut scripts
diff working/usr/lib/dracut/modules.d/95iscsi/parse-iscsiroot.sh \
     failing/usr/lib/dracut/modules.d/95iscsi/parse-iscsiroot.sh

Analyzing Console Logs

When investigating boot failures from kola tests:

# Extract kola artifacts
tar -xf kola-upgrade-<arch>.tar.xz

# Find console logs
find . -name "console.txt"

# Look for first error (not cascade failures)
grep -n -E "FATAL|failed to|Failed to|error:" console.txt | head -10

# Check module loading
grep -E "modprobe|insmod|systemd-modules-load" console.txt

Reference

Keep looking

Skills are one crate of 610,677. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.