# Edge Financial – Incident Report: Incoming Calls Failed on 2026-07-20

> **Date of Incident:** Monday 2026-07-20  
> **PBX Hostname:** edge-financial  
> **PBX Version:** V-Connect v1.1.185-3-ge64c11f  
> **Asterisk Version:** 22.4.1  
> **SIP Trunk:** ECN123 → `sip:154.119.162.123:5060` (account `4427301421`)  
> **Investigation Date:** 2026-07-21 10:10 SAST  
> **Investigated By:** Cascade (AI assistant) at request of Stuart Kemp  

---

## 1. Problem Statement

The client reported they were unable to receive incoming calls on Monday 2026-07-20. An investigation was requested to check for AMI leaks, Asterisk errors, and any other issues.

---

## 2. Key Findings

### 2.1 No AMI Leak

```
Username         IP Address           Start       Elapsed     FileDes   HttpCnt   ReadPerms   WritePerms
0 users connected.
```

- **0 AMI connections** at time of investigation.
- 2 AMI users configured (`admin`, `dialer`) — neither was connected.
- **No AMI leak detected.**

### 2.2 Two Unplanned Reboots on Jul 20

The system rebooted twice during the day. Both appear to be **hard power-offs** — no graceful shutdown was logged in the journal; the logs simply end with normal cron activity.

| Boot | Start Time | End Time | Duration | How it Ended |
|------|-----------|----------|----------|-------------|
| -2 | Sun Jul 19 17:14 | **Mon Jul 20 11:41** | ~18 hours | Hard power-off (no shutdown logged) |
| -1 | Mon Jul 20 11:42 | **Mon Jul 20 14:17** | ~2.5 hours | Hard power-off (no shutdown logged) |
| 0 (current) | Mon Jul 20 14:17 | Still running | ~20+ hours | N/A |

Evidence from `journalctl --list-boots`:
```
-2 de298f430fa84387ab246ac583ba3d86  Sun 2026-07-19 17:14:36 SAST  Mon 2026-07-20 11:41:02 SAST
-1 3349b3256014461f8637bee20e7c936c  Mon 2026-07-20 11:42:50 SAST  Mon 2026-07-20 14:17:02 SAST
 0 ead33884bd7a4a75a34018b216595f6c  Mon 2026-07-20 14:17:46 SAST  Tue 2026-07-21 10:15:19 SAST
```

Evidence from `last -x reboot`:
```
reboot   system boot  6.1.0-49-amd64   Mon Jul 20 14:17   still running
reboot   system boot  6.1.0-49-amd64   Mon Jul 20 11:42   still running
reboot   system boot  6.1.0-49-amd64   Sun Jul 19 17:14   still running
```

No corresponding `shutdown` entries exist for Jul 20, confirming these were **not** graceful shutdowns. The last recorded graceful shutdown was Mon Jun 22.

### 2.3 ECN123 Trunk Lost Registration and Failed to Recover

#### Phase 1: Trunk Unreachable (13:59–14:13, boot -1)

After the first reboot (11:42), Asterisk came up and was running. However at **13:59**, outgoing calls started failing because the ECN123 trunk was unreachable:

```
[2026-07-20 13:59:22] ERROR res_pjsip.c: Endpoint 'ECN123': Could not create dialog to invalid URI 'ECN123'.
                       Is endpoint registered and reachable?
[2026-07-20 13:59:22] ERROR chan_pjsip.c: Failed to create outgoing session to endpoint 'ECN123'
[2026-07-20 13:59:22] NOTICE app_dial.c: Unable to create channel of type 'PJSIP' (cause 3 - No route to destination)
```

This pattern repeated 4 times between 13:59:22 and 14:13:47. Each was a user attempting an outgoing call that failed.

#### Phase 2: Second Reboot and Registration Failure (14:17–14:34, boot 0)

At **14:17** the system rebooted again (hard power-off). When Asterisk started, it initially could not connect to MySQL:

```
[2026-07-20 14:17:57] WARNING res_odbc.c: res_odbc: Error SQLConnect=-1 errno=2002
    [unixODBC][ma-3.1.15]Can't connect to local server through socket '/var/run/mysqld/mysqld.sock' (2)
```

This ODBC error appeared **5 times** during startup, indicating MariaDB was not yet ready when Asterisk tried to connect. This is a **race condition** in the systemd startup order.

Asterisk then attempted to register with the ECN carrier at `154.119.162.123:5060`. The registration failed **10 consecutive times** over 16 minutes, with no response from the carrier:

```
[2026-07-20 14:18:36] WARNING: No response received from 'sip:154.119.162.123:5060' on registration attempt
                       to 'sip:4427301421@154.119.162.123', retrying in '60'
[2026-07-20 14:20:08] WARNING: No response received ... retrying in '60'
[2026-07-20 14:21:40] WARNING: No response received ... retrying in '60'
[2026-07-20 14:23:12] WARNING: No response received ... retrying in '60'
[2026-07-20 14:24:44] WARNING: No response received ... retrying in '60'
[2026-07-20 14:26:16] WARNING: No response received ... retrying in '60'
[2026-07-20 14:27:48] WARNING: No response received ... retrying in '60'
[2026-07-20 14:29:20] WARNING: No response received ... retrying in '60'
[2026-07-20 14:30:52] WARNING: No response received ... retrying in '60'
[2026-07-20 14:32:24] WARNING: No response received ... retrying in '60'
[2026-07-20 14:33:56] WARNING: Maximum retries reached when attempting outbound registration to
                       'sip:154.119.162.123:5060' with client 'sip:4427301421@154.119.162.123',
                       stopping registration attempt
```

After **maximum retries were reached**, Asterisk **gave up registering** entirely. This means:
- **No outgoing calls** via the trunk
- **No incoming calls** from the carrier (carrier had no active registration to route calls to)

### 2.4 Zero Calls Processed on Jul 20

The entire `full.1` log (covering all of Jul 20, 7042 lines) contains:
- **Zero** `Dial()` commands
- **Zero** incoming channel entries (`ECN123-`)
- **Zero** `trunks-in` or `from-trunk` context entries
- Only content is: startup messages, BLF subscription attempts, cron health-check UNIX socket connections, and the errors above

**No calls were made or received for the entire day.**

### 2.5 BLF Subscription Errors (Non-Critical)

Extension `200` has BLF keys configured for extensions that do not exist in `from-internal`:

```
NOTICE res_pjsip_exten_state.c: Endpoint '200' state subscription failed:
  Extension '201' does not exist in context 'from-internal' or has no associated hint
  Extension '202' does not exist in context 'from-internal' or has no associated hint
  Extension '203' does not exist in context 'from-internal' or has no associated hint
  Extension '223' does not exist in context 'from-internal' or has no associated hint
  Extension '226' does not exist in context 'from-internal' or has no associated hint
  Extension '230' does not exist in context 'from-internal' or has no associated hint
  Extension '233' does not exist in context 'from-internal' or has no associated hint
  Extension '235' does not exist in context 'from-internal' or has no associated hint
  Extension '20209' does not exist in context 'from-internal' or has no associated hint
```

These fire every ~2 minutes and are not causing call failures, but they add noise to the logs.

### 2.6 Other Startup Errors (Non-Critical, Pre-Existing)

These appear on every Asterisk startup and are pre-existing configuration issues, not related to the Jul 20 incident:

| Error | Notes |
|-------|-------|
| `codec_g729a.so: cannot open shared object file` | G.729 codec module missing |
| `Error opening directory /var/lib/asterisk/moh/Music_Test` | MOH directory doesn't exist |
| `Error opening directory /var/lib/asterisk/moh/New_MOH` | MOH directory doesn't exist |
| `res_ari_*.so missing dependency: res_ari` | ARI modules can't load (disabled) |
| `res_pjsip_publish_asterisk.c: Entity ID is not set` | Cluster entity ID not configured |
| `res_monitor.so: undefined symbol: ast_channel_monitor` | Legacy module incompatible |
| `PostgreSQL RealTime: Failed to connect` | Expected — using MariaDB, not PostgreSQL |
| `LDAP: No directory URL or host found` | Expected — LDAP not configured |

---

## 3. Current Status (as of 2026-07-21 10:15 SAST)

| Item | Status |
|------|--------|
| **System uptime** | 19 hours 56 minutes (since Jul 20 14:17) |
| **Asterisk** | Running (PID 550) |
| **MariaDB** | Running |
| **ECN123 trunk** | **Registered**, Avail, RTT 30ms |
| **Active channels** | 6 (3 calls in progress) |
| **Calls processed today** | 51 |
| **AMI connections** | 0 (no leak) |
| **SIP contacts** | 14 endpoints, all Reachable |
| **Memory** | 2.1Gi used / 23Gi total |
| **Disk** | 91G used / 938G total (11%) |
| **Load average** | 0.41, 0.46, 0.40 |

The system is currently healthy and processing calls normally.

---

## 4. Root Cause

**Power outages** at the client site caused two hard reboots. After the second reboot:

1. Asterisk started before MariaDB was ready (ODBC connection errors).
2. The SIP trunk registration to ECN (`154.119.162.123`) received **no response** for 10 consecutive attempts over 16 minutes.
3. Asterisk hit the maximum retry limit and **permanently stopped registration attempts** for the trunk.
4. With no active trunk registration, **all incoming and outgoing calls failed** for the remainder of the day.
5. The trunk eventually re-registered (likely via a PBX cron job or manual intervention), and calls are working today.

---

## 5. Recommendations for Developers

### 5.1 HIGH — Automatic Trunk Re-Registration After Max Retries

**Problem:** When `pjsip_outbound_registration` hits `max_retries`, it gives up permanently. There is no automatic recovery mechanism.

**Suggested fix:** Implement a cron job or watchdog script that:
1. Checks trunk registration status periodically (e.g., every 5 minutes)
2. If a trunk shows `Unregistered` or has no contact, forces a re-registration:
   ```bash
   asterisk -rx "pjsip send unregister ECN123"
   sleep 2
   asterisk -rx "pjsip send register ECN123"
   ```
3. Alternatively, increase `max_retries` in the PJSIP registration config or set it to `0` (unlimited retries with backoff).

Check the current `max_retries` value:
```bash
asterisk -rx "pjsip show registration ECN123" | grep -i retry
```

### 5.2 HIGH — Systemd Startup Order: Asterisk Should Wait for MariaDB

**Problem:** Asterisk starts before MariaDB is ready, causing ODBC `SQLConnect` failures during startup.

**Suggested fix:** Ensure the Asterisk systemd unit has proper ordering:
```ini
# /etc/systemd/system/asterisk.service.d/after-mariadb.conf
[Unit]
After=mariadb.service
Requires=mariadb.service
```

Or add a startup delay/readiness check in the existing drop-in config.

### 5.3 MEDIUM — UPS Recommendation for Client

**Problem:** Two hard power-offs in one day. No graceful shutdowns logged.

**Action:** Confirm with the client whether they have a UPS. If not, recommend one. If they do, check if it's functioning correctly and configured for graceful OS shutdown.

### 5.4 LOW — Clean Up BLF Subscriptions on Extension 200

**Problem:** Extension `200` has BLF keys for 9 extensions that don't exist (201, 202, 203, 223, 226, 230, 233, 235, 20209). This generates NOTICE-level log entries every ~2 minutes.

**Action:** Either:
- Create the missing extensions with hints, or
- Remove the BLF speed-dial keys from the phone provisioning for extension 200

### 5.5 LOW — Clean Up Pre-Existing Startup Errors

- Remove or fix the `codec_g729a.so` reference if G.729 is not licensed
- Create the missing MOH directories (`Music_Test`, `New_MOH`) or remove them from `musiconhold.conf`
- Disable loading of unused modules (`res_ari`, `res_monitor`, `res_hep_pjsip`, etc.) in `modules.conf`

---

## 6. Verification Commands

Developers can use these to check the current state at any time:

```bash
# Check trunk registration
asterisk -rx "pjsip show registrations"

# Check trunk endpoint status and active calls
asterisk -rx "pjsip show endpoint ECN123"

# Check all contacts reachability
asterisk -rx "pjsip show contacts"

# Check AMI connections
asterisk -rx "manager show connected"

# Check active calls
asterisk -rx "core show channels"

# Check for registration errors in log
grep -iE "No response|Maximum retries|Unreachable|Reachable" /var/log/asterisk/full | tail -n 20

# Check system reboots
journalctl --list-boots | tail -n 5
last -x reboot | head -n 5

# Check systemd service ordering
systemctl list-dependencies asterisk.service
```

---

## 7. Timeline Summary

| Time (SAST) | Event |
|---|---|
| Sun Jul 19 17:14 | System boot (boot -2) |
| **Mon Jul 20 11:41** | **Hard power-off** (no shutdown logged) |
| Mon Jul 20 11:42 | System boot (boot -1), Asterisk starts |
| Mon Jul 20 13:59 | Outgoing calls fail — ECN123 trunk unreachable (4 failed calls) |
| **Mon Jul 20 14:17** | **Hard power-off** (no shutdown logged) |
| Mon Jul 20 14:17 | System boot (boot 0), Asterisk starts |
| Mon Jul 20 14:17 | Asterisk cannot connect to MySQL (ODBC errors) — MariaDB not ready yet |
| Mon Jul 20 14:18–14:34 | ECN123 registration fails 10 times — no response from carrier |
| Mon Jul 20 14:33 | **Maximum retries reached — registration stopped permanently** |
| Mon Jul 20 14:34+ | **All incoming and outgoing calls dead** |
| (Unknown) | Trunk re-registered (possibly via cron or manual intervention) |
| Tue Jul 21 10:10 | Investigation — system healthy, trunk registered, calls flowing |