Skill

Splunk On-Prem Upgrade Skill

Scope: the on-prem estate that feeds Splunk Cloud — deployment servers (DS), agent managers, intermediate forwarders (IF), heavy forwarders (HF) including DB Connect (DBX) hosts, and the endpoint universal forwarders (UF) they manage. The indexing/search tier is Splunk Cloud (Splunk-managed, not upgraded by us) — but its version is a hard input to every upgrade decision below.

Golden rules:

1. The backout artifact is a full pre-upgrade tarball of $SPLUNK_HOME, stored under /tmp on the target, taken while splunkd is stopped. Backout = stop, move aside, untar, start. It is created on EVERY node, EVERY time, and never deleted until soak is complete.
2. One environment, one node at a time. Rolling upgrade over an explicit target list. Never parallelise nodes; never infer targets the user didn't give.
3. Failure on any node = PAUSE the whole run. Report, hold position, await a human decision (fix / backout / abort). Never continue to the next node past a failure.
4. Production only executes inside its approved maintenance window (dev/test/pre-prod may run BAU hours). Confirm the window before touching any prod host — outside it, refuse. Change-record (CR) gating is out of scope for this agent for now.
5. Verification uses REST + bearer token / curl for host-level checks, and the agent's MCP connection to the appropriate Splunk Cloud environment for data-landing checks — never Basic auth, never docker exec patterns.
6. Narrate and gate. Announce what is about to happen before each node, show evidence after, and get explicit approval at every gate (see §0.3).

---

0. Environment model, access & rolling-loop guardrails

0.1 Environments and scheduling

Env	Window rule	Gate strictness
dev	BAU hours OK	Confirm start of run + on failure
test	BAU hours OK	Confirm start of run + on failure
pre-prod	BAU hours OK	Confirm start of run + per-node go/no-go
production	Maintenance hours ONLY	Per-node go/no-go, evidence shown at every gate

Hard scheduling guardrail: if asked to upgrade production outside the agreed maintenance window, refuse and say why — offer to schedule instead. Confirm the current time against the agreed window before touching any prod host. (Change-record / CR gating is out of scope for this agent for now — do not ask for or check a CR number.)

Promotion path is dev → test → pre-prod → production. Never start an env before the previous env's run is green and soaked (user can explicitly waive for dev/test, never for prod).

0.2 Access pattern — jump box

Targets are only reachable via a jump box. The agent already knows how to stage a package and run commands through it (scp to the jump box, hop to the target, checksum re-verified on the target before install) — no need to re-derive the ssh mechanics here. Guardrails on top of that baseline capability:

• Jump box host/user/key paths come from config/environment — never hardcoded, never echoed.
• Re-verify the package checksum ON THE TARGET before install. A corrupt hop is a real failure mode.
• Every remote command is shown to the user in the narration before it runs on pre-prod/prod.
• No rm -rf, no wildcard deletes over ssh — only the documented runbook steps.

0.3 The rolling loop (per environment)

Input from the user: environment name + explicit ordered target list. Never expand the list yourself (no "and presumably also…").

GATE 0  Show the plan: env, targets in order, versions from->to, window, backout summary.
        Await explicit "go".
For each target, in order:
  GATE N.1  Announce node N: hostname, role, what will happen. (prod/pre-prod: await "go")
  1. Pre-checks (§3) - show results table
  2. Drain / quiesce (§3.5)
  3. Create backout tarball (§4 step 2) - confirm size + path to user
  4. Upgrade (§4)
  5. Post-checks (§5) incl. browser verification (§5.6) - show results table
  GATE N.2  Node verdict: GREEN -> proceed to node N+1.
            NOT GREEN -> PAUSE. Report exactly what failed with output.
            Offer: (a) investigate/fix and re-verify, (b) backout this node (§6),
            (c) abort the run. Do NOTHING until the user chooses.
END     Environment summary: per-node results, deviations, CR actuals. Soak starts.

The loop state (env, node index, per-node status, tarball paths) is written to a run log under docs/upgrades/<env>-<date>.md as it progresses — if the session dies mid-run, the next session resumes from the log, not from memory.

---

1. Upgrade order (estate-wide)

Upgrade management plane before data plane; upstream receivers before the things that send to them.

0. Splunk Cloud stack version          (read-only input — check, don't touch)
1. Deployment server / agent manager   (DS version MUST be >= all its deployment clients)
2. Monitoring Console / license nodes  (if present)
3. Heavy forwarders                    (DBX hosts last within this class — extra checks)
4. Intermediate forwarders             (rolling, one at a time, drain via LB)
5. Universal forwarders (endpoints)    (last; can lag by design)

Why this order:

• A DS older than its clients can mis-serve bundles and cannot manage newer client features. Splunk's rule: DS version >= deployment client version. Upgrading the DS first keeps this true throughout the rollout.
• Forwarders should be <= the version of what they send to. IFs send to Cloud (Cloud is current, so rarely a blocker); UFs send to IFs — so IFs upgrade before the UF fleet.
• HFs parse data. Upgrading them mid-fleet changes parsing behaviour for everything downstream, so treat each HF as a mini data-plane change: canary, verify sourcetypes/line-breaking unchanged, then continue.

2. Version path planning

Do this ONCE per campaign, before any CR is raised.

1. Target selection. Prefer the latest maintenance release of the target train (e.g. 9.4.x latest, not 9.4.0). Never take a .0 release into production without a stated reason.
2. Supported upgrade path. Splunk publishes a version-specific matrix at help.splunk.com/.../<target-version>/upgrade-or-migrate-splunk-enterprise/how-to-upgrade-splunk-enterprise — find the current version in the "Your current version" column and read across to the direct-upgrade target(s); if the current version isn't listed, an intermediate hop through a version that IS listed is required. There is no fixed "N major trains" rule of thumb — pull the actual matrix for the actual target version every time, it is revised release to release.
3. Splunk Cloud compatibility — IF-only. Intermediate forwarders are the last hop before Splunk Cloud, so this check applies to IFs, not DS/HF (which forward to IFs, not directly to Cloud). Use the agent's MCP connection to the target Splunk Cloud environment to note the stack's version; the IF being upgraded must stay within Splunk Cloud's supported-forwarder-version matrix.
4. OS support. Each Splunk train drops older OS versions (e.g. Splunk 10 drops older RHEL/Ubuntu). Check platform requirements for the target against cat /etc/os-release on every host. An OS upgrade prerequisite discovered mid-window is a failed change.
5. App compatibility sweep. Inventory every installed app per host (GET /services/apps/local?output_mode=json), then check each against Splunkbase's compatibility field for the target version. Private apps: check for python2-only code, deprecated libs, web.conf endpoints.
6. DB Connect matrix (HF/DBX hosts). Three-way compatibility: Splunk version ↔ DBX version ↔ Java version. Pull the current DBX release notes / Splunkbase compatibility page for the exact Java requirement for the target DBX build — this has shifted across DBX major versions, never assume it from a prior campaign (as of DBX 4.3.x the app supports Splunk platform 9.0–10.5; the JRE path is configurable in DB Connect settings, not a fixed value). Also verify JDBC driver add-ons against the new DBX. If DBX must be upgraded too, it happens in the SAME window as that HF's platform upgrade, DBX after platform.
7. KV store engine. WiredTiger is required from Splunk Enterprise 9.0 onward (introduced as an option in 8.1; mmapv1 is deprecated). Any DS/HF still pre-9.0 and on mmapv1 MUST migrate before/during the jump to 9.0+. On every full-Enterprise instance (DS, HF): GET /services/kvstore/status → check storageEngine. If mmapv1, run splunk migrate kvstore-storage-engine --target-engine wiredTiger (own mini-change, before the version upgrade).
8. Python runtime. Splunk Enterprise 9.0 is the first fully Python-3-only release — Python 2 was removed from all Splunk Enterprise/Cloud releases from September 2021 onward, so any HF still below 9.0 with Python-2-only scripted inputs is a blocker crossing into 9.0+. Default interpreter is 3.7 on 9.0–9.2, 3.9 from 9.3 onward. Scripted/modular inputs on HFs must run under python3 (python.version = python3 in inputs/commands stanzas).
9. TLS/OpenSSL. Splunk 9.4+/10 move to OpenSSL 3: weak-digest certs (MD5/SHA1) and TLS < 1.2 are rejected. Audit server.conf/inputs.conf/outputs.conf ssl stanzas and cert chains (openssl x509 -in cert.pem -noout -text | grep 'Signature Algorithm') BEFORE the window.

Deliverable of this phase: a version-path document (target version per component class, hop plan if needed, app/DBX/Java actions, blockers) committed to the repo under docs/.

3. Pre-checks (T-1 to T-0, per host)

Run all of these and record outputs. A failed pre-check is a STOP, not a note.

3.1 Scope confirmation

• Environment, ordered target list, and version target confirmed explicitly by the user — never inferred or expanded.
• For production: current time confirmed to be inside the agreed maintenance window.
• Change-record (CR) gating is out of scope for this agent for now — do not ask for or check a CR number.

3.2 Health baseline — the two questions that matter most

Before anything else, answer these two in plain terms, with evidence shown to the user:

1. Is this node sending its internal logs downstream right now? Prove it from the RECEIVING side (Splunk Cloud), not the node itself: index=_internal host=<host> earliest=-15m | stats count by sourcetype. Zero results = the pipe is already broken before you touch anything — STOP and fix that first; an upgrade will not fix a dead forwarder.
2. Any errors right now, before the upgrade? index=_internal host=<host> log_level=ERROR earliest=-24h | stats count by component — this is the baseline you diff post-upgrade against. New error components after the upgrade that weren't in this baseline are real findings; errors that already existed aren't upgrade regressions.

Supporting evidence (capture for the record, secondary to the two above):

# All via bearer token; substitute $H='Authorization: Bearer <token>'
curl -sk -H "$H" https://<host>:8089/services/server/info?output_mode=json          # version, OS, guid
curl -sk -H "$H" https://<host>:8089/services/server/health/splunkd/details?output_mode=json
curl -sk -H "$H" https://<host>:8089/services/apps/local?output_mode=json&count=0   # app inventory
curl -sk -H "$H" https://<host>:8089/services/kvstore/status?output_mode=json       # DS/HF only

• DS only: curl -sk -H "$H" https://<ds>:8089/services/deployment/server/clients?output_mode=json&count=0 → record client count. This must match post-upgrade.
• HF/DBX only: record max(rising column checkpoint) per enabled input — inputs keep running through the upgrade (§3.5), this is purely a baseline to diff post-upgrade, not a trigger to disable anything.

3.3 System readiness

• Disk: free space >= 2x the install package size on the install volume AND >= 20% free overall.
• ulimit -n >= 64000, ulimit -u >= 16000 for the splunk user; THP disabled (cat /sys/kernel/mm/transparent_hugepage/enabled → [never]).
• NTP in sync; OS version supported by target (from §2.4).
• Package downloaded and checksum verified (sha512sum -c) — never install an unverified package.

3.4 Backups

The primary backout artifact is the full $SPLUNK_HOME tarball taken in §4 step 2 (after splunk stop, immediately before install). It captures binaries, etc/, splunk.secret, KV store files, and DBX checkpoints in one consistent unit. Everything below is belt-and-braces on top of it, taken pre-window:

• $SPLUNK_HOME/etc tarball copied OFF the host (to the jump box staging area) — survives host loss.
• $SPLUNK_HOME/etc/auth/splunk.secret explicitly — without it, every encrypted credential (incl. all DBX identities) is unrecoverable.
• KV store: splunk backup kvstore (DS/HF) — restorable independently if only the KV store is damaged.
• DS: serverclass.conf + entire deployment-apps/ tree.
• HF/DBX: db_connections.conf, db_inputs.conf, identities.conf; verify db_connect_* collections exist in the KV backup (rising-column checkpoints live there).
• Disk space check for the tarball itself: free space on /tmp >= size of $SPLUNK_HOME (du -sh $SPLUNK_HOME).

3.5 Drain (window start)

• IF: remove from load balancer, then wait for output queues to flush: group=queue name=tcpout* current size → 0. Verify downstream still healthy on remaining IFs before proceeding.
• HF/DBX: no input-disable step. DBX inputs stay running until splunk stop in §4 — a clean stop interrupts a scheduled input the same way any restart would, and the checkpoint diff in §5.4 is what proves nothing was corrupted, not a pre-emptive disable.
• DS: nothing to drain, but note that clients failing phone-home during the window is expected — they retry.

4. Implementation runbook (per host)

# 1. Stop cleanly and verify
$SPLUNK_HOME/bin/splunk stop
ps -ef | grep splunkd          # must be empty (no orphaned mongod/python either)

# 2. Create the BACKOUT TARBALL - full SPLUNK_HOME, splunkd stopped, this IS the backout plan
tar -czf /tmp/splunk-preupgrade-$(hostname)-$(date +%Y%m%dT%H%M).tar.gz -C $(dirname $SPLUNK_HOME) $(basename $SPLUNK_HOME)
ls -lh /tmp/splunk-preupgrade-*.tar.gz   # confirm size is sane; report path+size to the user

# 2b. Verify the package checksum ON THE TARGET (it travelled jump box -> target)
sha512sum -c /tmp/splunk-<ver>.tgz.sha512

# 3. Install new version OVER the old
#    RPM:   rpm -U splunk-<ver>.x86_64.rpm
#    DEB:   dpkg -i splunk-<ver>-amd64.deb
#    tar:   tar -xzf splunk-<ver>.tgz -C /opt      (extracts over /opt/splunk)
chown -R splunk:splunk $SPLUNK_HOME

# 4. First start runs migration - do it in the foreground, read the output
sudo -u splunk $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes --no-prompt

# 5. Check migration results
ls $SPLUNK_HOME/var/log/splunk/migration.log.*
grep -iE "error|fail" $SPLUNK_HOME/var/log/splunk/migration.log.* || echo "migration clean"

# 6. Confirm version
$SPLUNK_HOME/bin/splunk version

Windows UF fleet (if any): msiexec /i splunkuniversalforwarder-<ver>.msi AGREETOLICENSE=Yes /quiet /norestart — same-directory install upgrades in place; config preserved.

UF fleet at scale: the DS deploys apps/config, not Splunk binaries. Binary upgrades of UFs go through OS package management (Ansible/SCCM/yum repo). Plan that channel; don't assume the DS can do it.

Component-specific additions:

• DS / agent manager: after start, splunk reload deploy-server and confirm no serverclass errors in splunkd.log.
• HF/DBX: after platform is verified, upgrade the DBX app if the matrix requires it (install new DBX package via REST app install, restart), verify Java (java -version matches DBX requirement, JAVA_HOME in dbx settings still valid), THEN re-enable inputs one at a time (§5.4).
• IF: re-add to LB only after §5 post-checks pass on the node.

5. Post-checks (per host, same window)

5.1 Platform

• splunk version / GET /services/server/info = target version.
• splunk btool check --debug → no new errors vs baseline.
• Health report green: GET /services/server/health/splunkd/details — every feature green (or matching pre-upgrade state).
• grep ERROR $SPLUNK_HOME/var/log/splunk/splunkd.log since restart — triage anything new vs the §3.2 baseline. Pay attention to SSL, X509, KVStore, Migration errors.
• KV store (DS/HF): GET /services/kvstore/status → ready, storageEngine: wiredTiger.

5.2 Data flow

• group=tcpout in metrics.log shows connections re-established to the correct destinations.
• On the DOWNSTREAM side, via the agent's MCP connection to the appropriate Splunk Cloud environment: index=_internal host=<upgraded-host> earliest=-15m | stats count > 0 — the node's own internal logs arriving proves the pipe end-to-end.
• Queue fill %s within baseline range (no new sustained back-pressure).
• Throughput (kb/s) within ~±20% of baseline for comparable time-of-day.

5.3 Deployment server specific

• Client count on /services/deployment/server/clients returns to baseline within 2 phone-home intervals.
• Push a no-op test: touch a comment in one deployment app, splunk reload deploy-server, confirm a canary client pulls the new checksum.

5.4 DB Connect specific (the highest-risk surface)

• DBX task server (Java process) running; DBX health dashboard loads.
• Each connection validates (test connection / | dbxquery connection=<c> query="SELECT 1" where the DB dialect allows).
• Inputs were never disabled (§3.5) — confirm each resumes on its next scheduled run and compare max(rising column) and event counts against the §3.2 baseline:
  • checkpoint jumped backwards → duplicates incoming: STOP, back out (§6).
  • checkpoint absent/reset → input will re-ingest from initial value: STOP, do not let it run to completion.
• dbx.log clean of driver/classpath errors (JDBC drivers can break on Java or DBX bumps).
• Scheduled DBX jobs fire on their next cron tick.

5.5 Browser verification — UI is actually up (mandatory, not optional)

REST proves the daemon; this proves the SERVICE. The agent has a browser-automation skill/agent available for this — use it rather than a raw curl check. This is a real step in the runbook, a node is NOT green without it:

1. Navigate to https://<fqdn>:8000. Confirm it actually renders — no blank page, no "Splunk is starting up", no cert error beyond the known self-signed state.
2. Log in with the designated account (creds from the env/secret store — never typed into chat, never logged). Confirm you land on the home/launcher page, not an error page.
3. Read the bulletin (Messages icon) — compare against the pre-upgrade baseline; new warnings are findings, not noise.
4. Confirm the version shown in the UI footer / Help > About matches the target version.
5. Run a sanity search: index=_internal | head 5 — confirm real events render in the results table (proves search head + local indexing path together).
6. Open the component page: DS → Settings > Forwarder Management (client list renders, counts sane); HF/DBX → DB Connect app opens, health page green, connections list intact.
7. Screenshot each step into the run evidence under docs/upgrades/. UFs have no web UI — skip this section for UF-class nodes.

5.6 Soak + close

• 24h soak per wave: re-run §5.1–5.4 spot checks, compare license usage and error counts to baseline.
• Only after soak is clean: proceed to next wave / close the CR with actual times and results.

6. Backout — revert the pre-upgrade tarball

Decision gate: the CR window end is the hard backout deadline — if post-checks aren't green with enough time left to back out, back out. Backout only ever targets the failed node; nodes already green stay upgraded (the run is PAUSED, not reversed).

$SPLUNK_HOME/bin/splunk stop
ps -ef | grep splunkd                      # verify fully stopped (no mongod/python orphans)
mv $SPLUNK_HOME ${SPLUNK_HOME}.failed-$(date +%Y%m%dT%H%M)     # keep for root-cause
tar -xzf /tmp/splunk-preupgrade-<host>-<stamp>.tar.gz -C $(dirname $SPLUNK_HOME)
chown -R splunk:splunk $SPLUNK_HOME
sudo -u splunk $SPLUNK_HOME/bin/splunk start
$SPLUNK_HOME/bin/splunk version            # must equal the ORIGINAL version

Then:

1. Run the FULL §5 post-check suite (incl. browser check) against the restored version — a backout is not done until it is verified.
2. DBX: checkpoints are inside the tarball so they restore consistently, but still diff max(rising column) vs the §3.2 record before re-enabling inputs.
3. IF: re-add to the LB only after verification.
4. Update the CR: backout executed at HH:MM, evidence attached. The run stays PAUSED.
5. Keep the .failed-* directory for root-cause; delete only after analysis.

After any backout: root-cause in the lab (Docker stack mirrors the version pair) before re-attempting. Never re-run a failed upgrade "to see if it works this time".

7. Fleet campaign pattern

Wave 0  Lab rehearsal        - replicate version pair in Docker, run the full runbook once
Wave 1  DS/agent manager     - single window; estate management restored before anything else moves
Wave 2  HF canary            - the LEAST critical HF, full soak; DBX host is never the canary
Wave 3  HF remainder         - incl. DBX hosts, one per window
Wave 4  IF rolling           - one node per iteration, drain->upgrade->verify->re-add; fleet stays N-1 capable
Wave 5  UF fleet             - via package management, batched by criticality, DS confirms phone-home per batch

Track the campaign in a single estate table (host, class, current ver, target ver, wave, CR, status, soak result) committed to docs/ and updated after every window.

8. Common failure modes

Symptom	Likely cause	First move
splunkd starts, mongod loops	KV store engine/version mismatch	migration.log + kvstore/status; if pre-10 engine was mmapv1, you skipped §2.7
DBX identities all invalid	splunk.secret changed/lost	restore splunk.secret from backup, restart
DBX inputs re-ingest from scratch	checkpoints lost (KV restore missed, or inputs ran mid-upgrade)	restore KV backup; next time quiesce per §3.5
Forwarding stops, SSL errors in splunkd.log	OpenSSL 3 rejecting weak certs / TLS<1.2	§2.9 audit was skipped; reissue certs SHA-256, set sslVersions=tls1.2
DS clients stop phoning home	DS < client version, or pass4SymmKey/cert issue post-migration	check client splunkd.log side first
Scripted inputs dead on HF	python2 code under a python3-only runtime	python.version stanzas + 2to3 the script
UF fleet unchanged after "upgrade"	assumed DS pushes binaries	it doesn't — use the OS package channel (§4)