Network troubleshooting and dns

Network Troubleshooting and DNS

Overview

"It's always DNS" is a joke because it's almost true. Network debugging is a discipline of elimination: work the layers in order, prove or clear each one, and never jump to the application before the foundation is verified.

Work the stack bottom-up. Physical/link → IP → routing → firewall → transport (TCP/UDP) → DNS → TLS → HTTP. Each layer depends on the ones below. Testing layer 7 while layer 3 is broken wastes hours.

One variable at a time. Change one thing, re-test, record the result.

When to Use

• "The service is unreachable"
• DNS resolution failures or stale records
• TLS handshake errors, certificate problems
• Intermittent connectivity or latency complaints
• Firewall / security group questions
• Load balancer and reverse proxy debugging
• HTTP API issues (status codes, redirects, headers)

The Layer-by-Layer Workflow

1. LINK      ip link show                      → is the interface UP?
2. IP        ip addr show                      → do we have an address?
3. ROUTE     ip route show                     → do we have a gateway/default route?
4. GATEWAY   ping -c3 <gateway>                → can we reach the first hop?
5. EXTERNAL  ping -c3 1.1.1.1                  → can we route out? (IP, no DNS)
6. DNS       dig example.com                   → does name resolution work?
7. TRANSPORT ss -tlnp / nc -zv host 443        → is the port open/listening?
8. TLS       openssl s_client -connect h:443   → does the handshake complete?
9. HTTP      curl -sv https://host/            → what does the app say?

The single most diagnostic pair of tests:

ping -c3 1.1.1.1        # works? → IP and routing are fine
ping -c3 example.com    # fails? → it's DNS. It's always DNS.

DNS Debugging

dig example.com                  # basic lookup; read the ANSWER and status: NOERROR
dig @8.8.8.8 example.com         # bypass local resolver — is it a resolver problem?
dig example.com +trace           # full delegation path from root
dig -x 1.2.3.4                   # reverse lookup
dig example.com ANY / MX / TXT   # specific record types
dig example.com +short           # just the answer
resolvectl status                # what resolver is systemd-resolved actually using?
resolvectl flush-caches          # clear stub cache after fixing records
getent hosts example.com         # what the OS actually resolves (NSS, incl. /etc/hosts)

Key concepts you must apply:

• TTL matters. A record change can take the old TTL to propagate. When planning a migration, lower the TTL to 60–300s a day before, not during.
• NXDOMAIN vs SERVFAIL vs timeout are three different problems: the name doesn't exist, the authoritative server is broken, or the network path is.
• /etc/hosts and nsswitch.conf override DNS. When resolution "makes no sense," check getent hosts first.
• Search domains (resolv.conf search line) make ping api resolve as api.corp.example.com. Always test the FQDN with a trailing dot: dig api.corp.example.com.

TLS / Certificate Debugging

# What certificate is actually being served?
openssl s_client -connect example.com:443 -servername example.com </dev/null

# Expiry at a glance
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

# Full chain validation against local CA store
openssl s_client -connect example.com:443 -servername example.com -CApath /etc/ssl/certs

# Local certificate inspection
openssl x509 -in cert.pem -noout -text

The usual suspects, in order of frequency:

1. Expired certificate — check notAfter.
2. Wrong name — certificate CN/SAN doesn't match the hostname (SNI matters: always pass -servername).
3. Missing intermediate — server sends an incomplete chain; browsers often hide this, curl and Java don't.
4. Untrusted CA — internal CA not in the client's trust store.
5. Clock skew — client's clock is wrong; "certificate not yet valid" is the tell.

HTTP Debugging with curl

curl -sv https://api.example.com/health 2>&1 | less   # everything: DNS, TLS, headers
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com
curl -sI https://example.com                          # headers only (HEAD)
curl -sL https://example.com                          # follow redirects — always check -I first
curl --resolve example.com:443:10.0.0.5 https://example.com   # test a specific backend bypassing DNS
curl -H 'Host: example.com' http://10.0.0.5/          # test vhost on a raw IP

Read status codes as layers:

• 301/302/307/308 → redirect; follow with -L and see where you end up
• 401/403 → reached the app; auth problem, not a network problem
• 404 → reached the app; routing problem inside it
• 502/503/504 → the proxy/gateway works; the backend behind it doesn't
• connection refused → nothing listening on that port (or firewall REJECT)
• connection timed out → firewall DROP, routing black hole, or host down

Packet-Level Diagnosis (when everything else fails)

# Is traffic even arriving at this host?
sudo tcpdump -i any -nn port 443

# Follow a request end-to-end
sudo tcpdump -i any -nn host 10.0.0.5 and port 5432 -w db.pcap

# Path discovery (ICMP may be filtered; try TCP)
traceroute -n example.com
mtr -n example.com                 # live, per-hop loss/latency — better than traceroute

Firewall inspection:

sudo nft list ruleset              # nftables (modern)
sudo iptables -L -n -v             # legacy
sudo firewall-cmd --list-all       # firewalld
sudo ufw status verbose            # ufw
ss -tlnp                           # what is actually listening, and which process owns it

Cloud reminder: the host firewall is only one layer. Security groups, NACLs, and cloud firewalls sit outside the VM — a perfectly configured host can still be blocked upstream.

The Output Template

When reporting or handing off a network diagnosis, state:

### Network Diagnosis: <symptom>
- **Layer reached:** [e.g., TCP connects, TLS handshake fails]
- **Evidence:** [the exact commands and outputs that prove it]
- **Ruled out:** [layers tested and cleared: link, IP, routing, DNS...]
- **Root cause hypothesis:** [most likely cause + why]
- **Fix applied / proposed:** [change + how it was verified]

Common Rationalizations

Rationalization	Reality
"It works from my machine, so it's the server's fault"	Different path, different DNS view, different firewall rules. Your machine proves the app works, not that the network does. Reproduce from the failing side.
"DNS propagated, I checked once"	You checked one resolver. Check the resolver the failing client actually uses (resolvectl status), plus @8.8.8.8 and the authoritative server.
"Let's just disable the firewall"	You've removed isolation and learned nothing. Capture packets, read rule counters, add a targeted allow rule.
"The cert is fine, I renewed it last week"	Renewed ≠ deployed. The server may be serving the old cert from a stale process or a different vhost. Ask the server what it's serving with openssl s_client.
"Intermittent means it's not the network"	Intermittent is exactly what packet loss, MTU problems, and one-bad-replica-in-a-pool look like. Use mtr and per-endpoint tests, not single pings.
"curl works, so the app is fine"	curl from the same host bypasses load balancers, proxies, and auth layers. Test through the same path users take.

Red Flags

• Debugging by randomly restarting services and routers "to see if it helps"
• Nobody knows the TTL of the record being migrated
• /etc/hosts entries on servers that duplicate real DNS (drift waiting to happen)
• Firewall rules with no comments, no owner, and a fear of touching them
• Certificates renewed manually with no calendar/alert before expiry
• "Works over VPN but not over the office network" and nobody has compared the paths

Verification

After any network fix:

• The original failing command now succeeds from the originally failing location
• Each layer in the stack was individually cleared, with the commands recorded
• DNS changes verified against the authoritative server, not just one cache
• TLS: openssl s_client shows the correct cert, complete chain, and valid dates
• No firewalls were disabled — a specific rule change was made and documented
• The fix survives reconnection/restart (not a transient "it worked once")
• What was broken and why is written down (ticket, runbook, or postmortem input)