# Meyers PBX — Post-Deployment Verification Checklist

**Target version:** v1.1.199 (or later)
**Applies to:** meyers-motors (and any box getting this update)
**Purpose:** confirm the recording-import/overload fix and the related trunk +
AMI fixes are live and healthy after `sudo /var/www/html/scripts/pbx-update.sh`.

> Run these read-only checks after the update. Each has an **Expected** result.
> Anything that doesn't match is flagged with what to do.

---

## 0. Deploy the update

```bash
sudo /var/www/html/scripts/pbx-update.sh
sudo systemctl restart apache2          # mod_php: load new code in the web layer
```

Confirm the version:

```bash
cd /var/www/html && git describe --tags
# Expected: v1.1.199 (or later)
```

> No Asterisk restart is required for this update. Only restart Asterisk during a
> quiet window if you specifically need to (e.g. to clear pre-fix orphans).

---

## 1. Recording import overload fix (the 2026-07-22 incident)

### 1.1 The runaway cron is gone
```bash
crontab -l | grep -i recording-sync
ls -la /etc/cron.d/recording-sync 2>/dev/null
```
- **Expected:** no output from either (the legacy every-minute cron is retired).
- If a line still exists: re-run `sudo -u www-data php /var/www/html/artisan pbx:deploy`
  (the "Retire legacy recording cron" step removes it), or remove it manually.

### 1.2 Import now runs via the scheduler with overlap protection
```bash
sudo -u www-data php /var/www/html/artisan schedule:list | grep -i recordings
```
- **Expected:** two entries, both `*/5 * * * *` — `recordings:process` and
  `recordings:import`.

### 1.3 Only ever ONE import running (never stacks)
```bash
ps aux | grep -E "recordings:import|recording-sync" | grep -v grep | wc -l
```
- **Expected:** `0` or `1`. (During the incident it was 11.)

### 1.4 Import is fast now (set-based + indexed)
```bash
time sudo -u www-data php /var/www/html/artisan recordings:import
```
- **Expected:** completes in a couple of seconds; prints `Imported N recording(s).`
  (Even with tens of thousands of files — it no longer scans per-file.)

### 1.5 The filename index exists
```bash
mysql -uroot asterisk -e "SHOW INDEX FROM call_recordings WHERE Column_name='filename';"
# (or: mysql -uasteriskuser -p ... if root has no socket auth)
```
- **Expected:** a row for `idx_call_recordings_filename`.

### 1.6 System load is healthy
```bash
uptime
nproc
```
- **Expected:** 1-minute load average comfortably **below** the CPU count
  (`nproc`). During the incident it was ~6 on a 4-core box.

### 1.7 No "call limit reached" rejections
```bash
grep -c "Maximum loadavg limit" /var/log/asterisk/full
grep "call limit reached" /var/log/asterisk/full | tail -3
```
- **Expected:** none since the update. If new ones appear, the box is genuinely
  overloaded — check `top`/`htop` for the culprit before touching `maxload`.

### 1.8 Recordings are still importing (spot check)
```bash
sudo -u www-data php /var/www/html/artisan tinker --execute='echo \App\Models\CallRecording::whereDate("created_at", today())->count()." imported today\n";'
ls /var/www/html/storage/app/public/recordings/*.wav | wc -l
```
- **Expected:** a non-zero, growing count consistent with the day's calls.

---

## 2. Trunk registration auto-recovery (power-cut / max-retries fix)

### 2.1 Watchdog is scheduled
```bash
sudo -u www-data php /var/www/html/artisan schedule:list | grep -i trunk-register
```
- **Expected:** `* * * * * php artisan pbx:trunk-register-watchdog` (every minute).

### 2.2 Watchdog is a no-op while trunks are healthy (does not flap them)
```bash
sudo -u www-data php /var/www/html/artisan pbx:trunk-register-watchdog
asterisk -rx "pjsip show registrations"
```
- **Expected:** the command prints nothing (no "re-registered" lines) and the
  trunk(s) stay `Registered`. It only acts when a registration is actually down.

### 2.3 Asterisk waits for MariaDB on boot
```bash
systemctl show asterisk.service -p After | tr ' ' '\n' | grep -E "mariadb|mysql"
systemctl show asterisk.service -p ExecStartPre | grep -o "mysqladmin ping"
```
- **Expected:** `mariadb.service` present in `After`, and `mysqladmin ping` present
  in `ExecStartPre` (the bounded readiness wait). This prevents the cold-boot race
  where Asterisk starts before the DB and loads no trunk config.

---

## 3. Trunk auth integrity (outbound-call auth fix)

### 3.1 No password-less trunk auth (would silently break outbound)
```bash
sudo -u www-data php /var/www/html/artisan pbx:readiness-check 2>&1 | grep -iA1 "Trunk auth"
```
- **Expected:** no "NO password" issue. A clean run reports 0 issues for trunk auth.
- If it flags an auth object: re-save the trunk in the GUI (auto-heals) or run
  `sudo -u www-data php /var/www/html/artisan pbx:deploy`.

### 3.2 Trunk registered and reachable
```bash
asterisk -rx "pjsip show registrations"
asterisk -rx "pjsip show endpoint ECN" | grep -iE "Endpoint:|Contact:|Aor:"
```
- **Expected:** `Registered`, contact `Avail` with a sane RTT.

---

## 4. AMI / Stasis leak (should remain resolved)

```bash
asterisk -rx "core show taskprocessors" | grep -c "stasis/p:manager:core"
asterisk -rx "core show taskprocessors" | grep "stasis/pool-control"
ls -la /etc/asterisk/users.conf 2>/dev/null   # should NOT exist
```
- **Expected:** manager-sub count low and **flat** over time (not climbing);
  pool-control queue depth low at idle; `users.conf` absent (renamed to
  `users.conf.disabled` by the deploy — its presence caused the reload leak).

---

## 5. fail2ban / SIP brute-force protection

```bash
sudo fail2ban-client status asterisk-scanner | grep -iE "file list|banned"
sudo fail2ban-regex /var/log/asterisk/messages asterisk-pjsip | grep -iE "^Lines:"
```
- **Expected:** jails read `/var/log/asterisk/*` (file backend), and the pjsip
  filter **matches** lines (non-zero "matched"). Scanner tier bans non-existent-
  extension probes aggressively; auth tier is lenient and whitelist-exempt.

---

## 6. Overall health snapshot

```bash
asterisk -rx "core show channels count"      # active calls
asterisk -rx "pjsip show contacts"           # endpoints reachable
asterisk -rx "manager show connected"        # AMI connections (0 or the dialler only)
uptime                                       # load
df -h /                                       # disk
free -h                                       # memory
```

**Green board = ** load below `nproc`, trunk `Registered`, contacts `Avail`,
manager subs flat, no "call limit reached", one-or-zero import processes.

---

## 7. If something is wrong — quick triage

| Symptom | First check | Likely fix |
|---|---|---|
| "Service unavailable" / calls rejected | `uptime` + §1.7 | Find the load source in `top`; confirm §1.1–1.3 (no stacked imports) |
| No inbound/outbound after a reboot | §2.2 `pjsip show registrations` | Watchdog re-registers within ~60s; if not, `asterisk -rx "pjsip send register <trunk>"` |
| Outbound fails, trunk shows Registered | §3.1 readiness check | Password-less auth → re-save trunk / `pbx:deploy` |
| Manager subs climbing | §4 | Ensure `users.conf` absent; then one Asterisk restart at 0 calls to clear orphans |
| Import process count > 1 | §1.1 | Legacy cron still present — retire it (`pbx:deploy`) |

---

## Reference — what changed in this update

| Area | Change | Version |
|---|---|---|
| Recording import | Set-based (one query), bounded per run, batched | v1.1.199 |
| Recording import | Now scheduler-driven with `withoutOverlapping` (no stacking) | v1.1.199 |
| Recording import | Index on `call_recordings.filename` | v1.1.199 |
| Recording import | Legacy raw `recording-sync.sh` cron retired on deploy | v1.1.199 |
| Trunk registration | `pbx:trunk-register-watchdog` auto-recovers dropped registrations | v1.1.197 |
| Boot ordering | Asterisk waits for MariaDB (systemd drop-in) | v1.1.197 |
| Trunk auth | Reference-based self-heal + readiness check | v1.1.196 |
| Stasis leak | `users.conf` removed on deploy (per-reload leak) | v1.1.188 |