Site down at 2 a.m.: the first 15 minutes when you're the only one on call
Author: OpsMate team
Updated
Based on hands-on troubleshooting with OpsMate. Run commands only on servers you are authorized to access; replace paths and service names for your environment.
When the site is down at night and you're the only one on call, follow a fixed 15-minute triage order—do not reboot the whole server first. A reboot can wipe evidence and interrupt a database write. Order: confirm impact → is the machine alive → entry point and app → logs from the failure window only → write three lines before changing anything. Assumes a typical Linux VPS with systemd, Nginx and Docker; swap paths and ports. This page is first-pass triage; deeper steps are in the linked guides.
People look up the same outage as a VPS that died overnight, a server that suddenly shut down, or a question about how to troubleshoot. The first pass is the same either way: confirm whether it is down for everyone, then see whether the machine is still up. If you do not know Linux or SSH, start with the site is suddenly down and you don't know servers. If you can already log in, continue from step 1 below.
0. The site is down and you don't know servers: do these three things first
You can triage without Linux and without SSH. These three steps use a phone or your own computer. You do not need to type commands on the server:
- Switch networks: is it down for everyone?
Open it again on mobile data, and ask a colleague to try another network. If it fails only for you, check local DNS, a proxy, and cache before you touch the server or reboot. - Open the cloud console: is the instance still running?
Check whether the instance is Running, whether the public IP is still there, and whether the security group or firewall allows 80/443 (or your app port). If the instance is stopped, suspended for non-payment, or the security group blocks everything, neither a panel nor SSH can get in. Fix that in the console first, and use the console VNC or serial console if you need to. - Save evidence, then find someone or learn SSH.
Write down roughly when it started, who is affected, console screenshots (instance status and graphs), and whether the page shows a timeout, a 502, or a certificate error. If your own computer can run commands, also record the outside check (replaceexample.comwith your domain):
Run these on your own computer, not on the server. Save the status code andcurl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --max-time 10 https://example.com/ dig +short example.com Adigoutput as-is. Cover internal addresses and customer information before you share them.
After you confirm it is down for everyone and the instance is still running, continue from “1. Minutes 0–2” below if you can already SSH and log in (outside curl, then whether the machine is alive, then the entry point and the app, then logs from the failure window, then three lines before you change anything). If you cannot log in, take the evidence from these three steps to the person who built it, a per-incident consultant, or a colleague. That is more useful than saying only that the site is down.
For a one-page inventory of a server someone else built, see Inherited a server: what to check first. For the full answer when you cannot manage a server at all, see I can't manage a server. What should I do?.
1. Minutes 0–2: confirm the impact from outside
Run these on your own computer, not on the server. Replace example.com with your domain:
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --max-time 10 https://example.com/
dig +short example.com A
dig +short example.com AAAA
000or a timeout: no HTTP response came back at all. Think DNS, network, firewall or security group, TLS handshake, or an unreachable machine.502/504: the gateway (Nginx or a CDN) answered but got no valid response from upstream. Focus on the application; see Nginx 502 Bad Gateway.500: the request reached the application and the application failed. Go to its logs.200: it works from where you are. The problem may be limited to one region, one path or some users; ask which page and which network.digreturns an address you don't expect for your server or CDN: check whether DNS records changed or the domain expired.
What this does not tell you: one vantage point is not every user, and if a CDN sits in front, the status code may come from the CDN's error page rather than your origin. A second try over mobile data is a cheap cross-check.
2. Minutes 2–5: is the machine alive?
Try to SSH in. If SSH fails, go to your cloud provider's console: instance status, monitoring graphs, security group rules, and the web-based VNC or serial console if needed. In that situation OpsMate can't reach the machine either. If you can log in:
uptime
nproc
free -h
df -h
df -i
- Compare the load averages from
uptimewith the core count fromnproc. Load that stays well above the core count means work is queueing for CPU or disk I/O. - In
free -h, read theavailablecolumn, notused. Linux uses spare memory for cache, so highusedalone is not a memory shortage. - A filesystem at 100% in
df -h, or inodes at 100% indf -i, means writes fail, and databases and apps often fall over as a result. Go to Linux disk usage suddenly spikes and don't start deleting files yet.
What this does not tell you: which process is responsible, or what the peak looked like when things broke. These are snapshots of right now.
3. Minutes 5–8: are the entry point and the app running?
systemctl --failed --no-pager
systemctl status nginx --no-pager
sudo ss -ltnp
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
systemctl --failedlists units that failed. An empty list doesn't clear your app, because not every app runs under systemd.ss -ltnpshows what is listening on 80/443 and on your app's port (you need sudo to see process names). Nginx up but nothing on the app port is the classic 502 pattern.- A container in
RestartingorExitedindocker ps -a: follow Docker container keeps restarting.
Then hit the app directly from the server, bypassing DNS, CDN and Nginx. The port and path are examples; use your real upstream and a safe health-check path:
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 5 http://127.0.0.1:3000/
If this works locally but the site fails from outside, look at Nginx, the firewall, TLS certificates or DNS. If it fails locally too, start with the app. A listening port is not proof of a healthy app: it can accept connections and still hang or error on every request.
4. Minutes 8–12: read only the logs from the failure window
Ask yourself roughly when it started, then read around that time instead of scrolling from the top:
journalctl -p err --since "30 min ago" --no-pager | tail -n 100
sudo tail -n 100 /var/log/nginx/error.log
docker logs --since 30m --tail 200 --timestamps YOUR_CONTAINER
sudo dmesg -T | grep -i -E 'out of memory|killed process' | tail -n 20
journalctl -p errshows priority err and above. Many apps write errors to their own log files, so an empty result here proves little.- Your Nginx
error.logpath depends on your config. Lines likeconnect() failed (111: Connection refused)point at an upstream that isn't listening. - In
docker logs, look for the first error before the exit, not just the last line. ReplaceYOUR_CONTAINERwith the real name. Out of memory/Killed processindmesgmeans the kernel killed something at some point. It only counts as evidence if the timestamp lines up with the outage. If logs mention a full disk, Docker logs filling your disk may also apply.
Before you share or submit any of this output, redact passwords, tokens, connection strings, customer emails and internal addresses.
5. Minutes 12–15: write three lines before you touch anything
Before any restart or edit, write this down, in your notes or your team chat:
- Symptom: when it started, who is affected, what the outside
curlreturned. - Evidence: which command and which output line support your theory, and what is still unconfirmed.
- Smallest action + rollback: the one thing you plan to do, how you'll verify it worked, and how you'll undo it if it doesn't.
Common smallest actions. Each one is a change you decide on, not a diagnostic step:
- Restart only the one failing service or container, not the whole machine. A restart clears in-memory state, so copy the output from the steps above somewhere safe first.
- Before editing Nginx config, back up the original file, run
sudo nginx -tto check syntax, then plan the reload. If it goes wrong, put the backup back and reload again. - If the disk is full, remove only what you've confirmed is disposable; snapshot or back up business data first.
- If you deployed recently and the evidence points at the new code, rolling back to the previous release is usually safer than a 2 a.m. hotfix, as long as you know whether the database migration can be reversed.
- Leave a full reboot for last. Before rebooting, check whether
/var/log/journalexists. If it doesn't, the journal may live only in memory and this boot's logs will be gone after the reboot.
After the action, repeat the outside curl from step 1 and the resource checks from step 2, confirm recovery, and note the time.
Doing this in OpsMate
OpsMate puts an SSH terminal and an AI assistant on the same server page. You can type every command above straight into the terminal. When you can't remember the right command, describe the problem to the AI in plain language. The AI proposes troubleshooting commands, and beyond logs it can use ps, df, ss, docker and journalctl for checks. The commands and output stay in the terminal so you can check them and review them later. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first. When you're away from your desk, patrol checks and Telegram alerts can tell you about a problem first. Receiving Telegram alerts doesn't require storing credentials in the cloud; only running SSH on the server remotely from Telegram does. See Server alerts in Telegram.
The limits, plainly:
- If you can't SSH into the machine, neither can OpsMate. Start with your cloud provider's console.
- The AI is mainly for troubleshooting. Commands that fail the safety check are blocked and never run. Low-risk fixes run automatically by default, and every step is logged. The rollbacks and config edits in this guide are yours to decide on. For what OpsMate does and doesn't do on its own, see the FAQ.
- An AI summary is a starting point. Check it against the command output; it is not proof of a root cause.
- Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first. What the desktop app keeps on your own machine by default is your SSH credentials.
Illustrative example
Illustrative example (not a real customer case): at 2 a.m. the outside curl returns 502. SSH works and load is low. systemctl status nginx shows Nginx running, but ss -ltnp shows nothing listening on the app's port 3000. docker ps -a shows the app container stuck in Restarting, and the first error in docker logs is a failed database write. Back in df -h, the root filesystem is at 100%. The 502 is only the symptom; the disk is the problem. Follow the disk guide to find out whether logs or business data filled it, remove only what's confirmed disposable, then watch whether the container recovers. Rebooting the whole machine in minute one would have left the disk just as full, the outage would have come back a few minutes later, and there would have been less evidence to go on.
AI diagnostic prompt
The site stopped loading about 30 minutes ago. Without changing anything on this server, check load, available memory, disk space and inodes, failed systemd units, listening ports, Docker container status, and system, Nginx and application errors from the last 30 minutes. Separate confirmed facts from hypotheses and list what information is still missing. Do not restart services, delete files or change configuration. If a change is needed, propose the smallest step with its risks, how to verify it and how to roll it back, and wait for my approval.
FAQ
My website suddenly went down and I'm not technical. What should I do?
Check from outside first: open the site on your phone over mobile data and note the time and the exact error. If you or someone helping you can log in, check whether the machine is up, the disk is full and the app is running, and save that output before rebooting anything. Steps 1–3 above walk through exactly that.
The freelancer who set up our server left. How do I take over?
During an outage, the first question is who can still get in: the cloud console, an SSH key, the domain registrar. If nobody on your side can, the cloud provider's console and support are your way in. Once the site is back, take over the accounts and revoke the freelancer's access, and add the entry points, dependencies, and owners to the handover inventory so the next incident does not start from a blank page.
When the failure is a certificate warning
The page loads and the browser warns about the certificate, or the handshake fails. From your own computer, read the leaf the public endpoint sent. Do not force a renewal, and do not reload Nginx. If a CDN or load balancer terminates TLS, the public leaf and the origin leaf may differ. A different notAfter or issuer does not prove which one is old. *.example.com covers one label, such as api.example.com. It does not cover a.api.example.com. A timeout, an error, or a blank result is unknown.
timeout 15 openssl s_client -servername example.com -connect example.com:443 </dev/null | openssl x509 -noout -subject -issuer -serial -fingerprint -sha256 -dates -ext subjectAltName
echo "pipe_status:$?"
timeout 15 openssl s_client -servername example.com -connect example.com:443 </dev/null | openssl x509 -noout -checkhost example.com
echo "pipe_status:$?"
The s_client output is the leaf that endpoint sent. It does not prove a trusted chain or revocation status. In an ordinary shell, pipe_status 0 is only the exit status of x509 at the end of the pipe. It does not mean the TLS handshake, the trust chain, or the timeout check succeeded. -checkhost only checks the name. Replace ORIGIN_IP with an address you are allowed to use. Compare serial numbers and SHA-256 fingerprints only for the same domain, the same SNI, the same termination layer, and the same node:
timeout 15 openssl s_client -servername example.com -connect ORIGIN_IP:443 </dev/null | openssl x509 -noout -subject -issuer -serial -fingerprint -sha256 -dates -ext subjectAltName
echo "pipe_status:$?"
timeout 15 openssl s_client -servername example.com -connect ORIGIN_IP:443 </dev/null | openssl x509 -noout -checkhost example.com
echo "pipe_status:$?"
On the machine that terminates TLS, run sudo nginx -T and copy ssl_certificate from the server { } block whose server_name includes your domain. Put that path in PATH_FROM_THE_MATCHING_SERVER_BLOCK. nginx -T is the config on disk plus a config test, not proof of what the running process loaded. Redact the output. The timer and hooks are clues: read enabled state, status, exit result, and the renewal log. A missing certbot.timer can be snap, cron, or another client. Do not call that “not certbot.” Do not run the hooks, and do not reload by default.
sudo nginx -T
sudo certbot certificates
systemctl is-enabled certbot.timer
systemctl show certbot.timer -p ActiveState -p UnitFileState -p Result --no-pager
systemctl show certbot.service -p ActiveState -p Result -p ExecMainStatus --no-pager
sudo journalctl -u certbot.service -n 40 --no-pager
sudo ls -l /etc/letsencrypt/renewal-hooks/deploy /etc/letsencrypt/renewal-hooks/post
sudo openssl x509 -in PATH_FROM_THE_MATCHING_SERVER_BLOCK -noout -subject -issuer -serial -fingerprint -sha256 -dates -ext subjectAltName
sudo openssl x509 -in PATH_FROM_THE_MATCHING_SERVER_BLOCK -noout -checkhost example.com
Different fingerprints at the same endpoint let you suspect the process has not loaded that file. They are still not a reason to reload. If a condition is missing, write unknown. The fuller record is on the site is suddenly down. OpsMate does not renew certificates for you.
I run a small business and can't afford a sysadmin. How do I keep my server running?
Prepare before the night it breaks: an outside uptime alert that reaches your phone, a short weekly check of disk, memory and backups, and someone you can pay per incident. When it does break, run the 15-minute order on this page and hand over the evidence instead of guessing. The small-team VPS monitoring checklist lists what to monitor.
Try OpsMate
500 free AI calls per month and unlimited servers. The desktop app keeps your SSH credentials on your own machine by default.
References
- curl man page — confirm HTTP status and timing from outside
- journalctl(1) — read-only system logs for the failure window
- Module ngx_http_proxy_module — upstream proxy settings often involved in 502s
- df(1) — quick check when a full disk takes the app down
FAQ
The site is suddenly down and I don't know servers. What should I do?
Switch networks to see whether it is down for everyone, then open the cloud console and check whether the instance is Running and whether the security group allows traffic. Write down the time and what you see. These three steps do not need Linux commands. After you confirm it is down for everyone and the machine is still running, continue from step 1 on this page if you can SSH; if you cannot, take the evidence to someone who can. See “The site is down and you don't know servers: do these three things first” above.
Need help interpreting the evidence?
OpsMate (itops.sh) helps developers and operators investigate with AI, and you review the evidence yourself. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first.
Start free Desktop with local credentials