# PBX Server Optimisation Guide

This document covers all optimisations applied to the PBX server for call quality, security, and stability. Run the script below on every new server installation.

---

## Quick Start

```bash
sudo bash /var/www/html/scripts/pbx-server-optimise.sh
```

The script is **fully idempotent** — safe to re-run at any time.

---

## What the Script Does

### 1. Swap Space (minimum 4GB)

**Problem:** A VoIP server with exhausted swap causes the OS to page-fault during RTP processing, directly causing audio jitter and latency spikes.

**Fix:** Creates a 4GB swapfile and sets `vm.swappiness=10` so the OS only uses swap as a last resort.

```bash
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl -w vm.swappiness=10
```

**Persisted in:** `/etc/sysctl.d/99-pbx.conf`

---

### 2. Kernel UDP Socket Buffers

**Problem:** Default Linux UDP socket buffers (~208KB) are far too small for a busy RTP server. Under load, RTP packets are dropped at the kernel before Asterisk even sees them.

**Fix:** Increase to 16MB.

```bash
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.rmem_default=1048576
sysctl -w net.core.wmem_default=1048576
sysctl -w net.core.netdev_max_backlog=5000
sysctl -w net.ipv4.udp_rmem_min=8192
sysctl -w net.ipv4.udp_wmem_min=8192
sysctl -w net.ipv4.ip_local_port_range="10000 65000"
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.core.somaxconn=1024
```

**Persisted in:** `/etc/sysctl.d/99-pbx.conf`

---

### 3. Asterisk File Descriptor Limits

**Problem:** The Asterisk process soft limit of 1,024 open files means it can only handle ~500 simultaneous calls before failing to open new RTP sockets.

**Fix:** Raise to 65,536 via a systemd override drop-in. Also protect Asterisk from the OOM killer.

```ini
# /etc/systemd/system/asterisk.service.d/limits.conf
[Service]
LimitNOFILE=65536
LimitNPROC=8192
LimitCORE=infinity
OOMScoreAdjust=-100
```

Apply with:
```bash
systemctl daemon-reload
systemctl restart asterisk
```

Verify with:
```
asterisk -rx "core show settings" | grep "Maximum open"
# Should show: Maximum open file handles: 65536
```

---

### 4. Opus Codec

**Problem:** `res_format_attr_opus.so` was loaded (SDP negotiation only) but `codec_opus.so` (the actual encoder/decoder) was missing. WebRTC/WebPhone endpoints silently fell back to ulaw.

**Fix:** Build from source using the [traud/asterisk-opus](https://github.com/traud/asterisk-opus) open source module against the Asterisk source tree.

```bash
cd /usr/src
git clone https://github.com/traud/asterisk-opus.git
cp asterisk-opus/codecs/codec_opus_open_source.c asterisk-18.26.4/codecs/
cp asterisk-opus/codecs/ex_opus.h asterisk-18.26.4/codecs/
cp asterisk-opus/include/asterisk/opus.h asterisk-18.26.4/include/asterisk/
cd asterisk-18.26.4
make codecs
cp codecs/codec_opus_open_source.so /usr/lib/asterisk/modules/codec_opus.so
asterisk -rx "module load codec_opus.so"
```

Verify:
```
asterisk -rx "module show like codec_opus"
# codec_opus.so  Opus Coder/Decoder  Running
```

---

### 5. Asterisk Logger — Production Verbosity

**Problem:** Writing `debug` and `verbose` to `/var/log/asterisk/full` on every call causes excessive disk I/O that introduces scheduling jitter on busy servers.

**Fix:** `/etc/asterisk/logger.conf`

```ini
[logfiles]
full    => notice,warning,error,dtmf
console => notice,warning,error
messages => notice,warning,error
```

Apply with:
```bash
asterisk -rx "logger reload"
```

---

### 6. QoS / DSCP Marking on All Endpoints

**Problem:** All PJSIP endpoints had `tos_audio=0` and `cos_audio=0` — RTP packets were being sent as best-effort traffic, with no network prioritisation.

**Fix:** Applied to all 107 existing endpoints in the database and set as defaults in the application controllers for all new endpoints going forward.

| Parameter | Value | Meaning |
|---|---|---|
| `tos_audio` | `184` | DSCP EF (Expedited Forwarding) — highest RTP priority |
| `tos_video` | `136` | DSCP AF41 — video traffic |
| `cos_audio` | `5` | 802.1p priority for audio (layer 2) |
| `cos_video` | `4` | 802.1p priority for video (layer 2) |

Database patch:
```sql
ALTER TABLE ps_endpoints
  ADD COLUMN IF NOT EXISTS tos_audio varchar(10) DEFAULT NULL,
  ADD COLUMN IF NOT EXISTS tos_video varchar(10) DEFAULT NULL,
  ADD COLUMN IF NOT EXISTS cos_audio int(11) DEFAULT NULL,
  ADD COLUMN IF NOT EXISTS cos_video int(11) DEFAULT NULL;

UPDATE ps_endpoints SET
  tos_audio = '184', tos_video = '136',
  cos_audio = 5,     cos_video = 4,
  rtp_timeout      = CASE WHEN rtp_timeout IS NULL OR rtp_timeout = 0 THEN 60 ELSE rtp_timeout END,
  rtp_timeout_hold = CASE WHEN rtp_timeout_hold IS NULL OR rtp_timeout_hold = 0 THEN 300 ELSE rtp_timeout_hold END;
```

---

### 7. RTP Timeout Defaults

**Problem:** `rtp_timeout=0` meant hung/zombie calls (dropped without a SIP BYE) were never cleaned up, consuming channels indefinitely.

**Fix:** Set as defaults in `ExtensionController` and `TrunkController` for all new endpoints:

| Parameter | Value | Effect |
|---|---|---|
| `rtp_timeout` | `60` | Clear call if no RTP for 60s during active call |
| `rtp_timeout_hold` | `300` | Clear call if no RTP for 5 min while on hold |

---

## Results After Optimisation

| Metric | Before | After |
|---|---|---|
| Swap available | 0 bytes (exhausted) | 4.0 GB free |
| UDP receive buffer | 208 KB | **16 MB** |
| Swappiness | 60 | **10** |
| Asterisk file handles | 1,024 | **65,536** |
| Opus codec | Not loaded | **Loaded** |
| QoS marking | None (0) | **DSCP EF (184)** |
| Zombie call cleanup | Disabled | **60s / 300s** |
| Log verbosity | debug+verbose | **notice/warning/error** |

---

## Files Modified

| File | Change |
|---|---|
| `/etc/sysctl.d/99-pbx.conf` | UDP buffers, swappiness |
| `/etc/systemd/system/asterisk.service.d/limits.conf` | File descriptor limits |
| `/etc/asterisk/logger.conf` | Production log verbosity |
| `/usr/lib/asterisk/modules/codec_opus.so` | Opus codec installed |
| `app/Http/Controllers/ExtensionController.php` | QoS + RTP timeout defaults |
| `app/Http/Controllers/TrunkController.php` | QoS + RTP timeout defaults |
| Database `ps_endpoints` (107 rows) | QoS + RTP timeout backfilled |

---

---

## Phase 2 Optimisations (Applied Manually)

### 8. DB Indexes on PJSIP Realtime Tables

Every inbound SIP request (INVITE, REGISTER, OPTIONS) causes Asterisk to query `ps_endpoints` by `context` and `transport`. Without indexes these are full table scans.

```sql
ALTER TABLE ps_endpoints
  ADD INDEX IF NOT EXISTS idx_context   (context),
  ADD INDEX IF NOT EXISTS idx_transport (transport);

ALTER TABLE ps_aors
  ADD INDEX IF NOT EXISTS idx_qualify (qualify_frequency);
```

### 9. ODBC Connection Pooling

Default ODBC config has a single connection — all concurrent calls compete for it. Pool of 10 with health-check:

```ini
# /etc/asterisk/res_odbc.conf
[asterisk]
enabled        => yes
dsn            => asterisk
username       => asteriskuser
password       => <password>
pre-connect    => yes
pooling        => yes
limit          => 10
idlecheck      => 60000
sanitysql      => select 1
connect_timeout => 10
```

### 10. MariaDB InnoDB Buffer Pool

Increased from 128MB to 512MB so the entire Asterisk database fits in RAM:

```ini
# /etc/mysql/mariadb.conf.d/50-server.cnf — [mariadb-10.11] section
innodb_buffer_pool_size         = 512M
innodb_buffer_pool_instances    = 2
innodb_log_file_size            = 128M
innodb_flush_log_at_trx_commit  = 2
innodb_flush_method             = O_DIRECT
query_cache_type                = 1
query_cache_size                = 32M
query_cache_limit               = 2M
max_connections                 = 300
thread_cache_size               = 32
table_open_cache                = 400
```

> **Note:** Sorcery memory cache (`res_sorcery_memory_cache`) was tested but causes a segfault in Asterisk 18.26.4 with the realtime+ODBC backend combination on this build. DB indexes + ODBC pooling achieve similar reduction in DB load safely.

---

---

## Asterisk 22 Upgrade (from 18.26.4)

Asterisk 18 reached EOL October 2025. This server was upgraded to **Asterisk 22.4.1** (current LTS).

### Key differences in v22

- **Lazy realtime loading** — endpoints are fetched from ODBC on demand, not all pre-loaded at startup. `pjsip show endpoints` only shows actively registered endpoints — this is correct and more efficient.
- **ODBC pooling syntax changed** — `pooling`/`limit`/`idlecheck` replaced by `max_connections`.
- **ABI break** — any `.so` modules compiled against Asterisk 18 will segfault v22 on startup. All custom modules must be recompiled.

### Upgrade procedure

```bash
# 1. Backup current modules
mkdir -p /usr/lib/asterisk/modules.backup-18.26.4
cp /usr/lib/asterisk/modules/*.so /usr/lib/asterisk/modules.backup-18.26.4/

# 2. Apply FQDN NAT patch (Teams Direct Routing requirement)
cp /usr/src/asterisk-22.4.1/res/res_pjsip_nat.c /usr/src/asterisk-22.4.1/res/res_pjsip_nat.c.orig
# Edit lines ~368 and ~380 in res_pjsip_nat.c — replace:
#   ast_sockaddr_stringify_host(&transport_state->external_signaling_address)
# with:
#   (!ast_strlen_zero(transport->external_signaling_address)
#       ? transport->external_signaling_address
#       : ast_sockaddr_stringify_host(&transport_state->external_signaling_address))

# 3. Configure and build
cd /usr/src/asterisk-22.4.1
./configure --with-jansson-bundled --with-pjproject-bundled
make menuselect.makeopts
menuselect/menuselect --enable codec_opus --enable res_odbc --enable res_config_odbc \
  --enable res_pjsip --enable res_pjsip_nat --enable res_srtp \
  --enable res_http_websocket --enable res_timing_timerfd \
  --enable res_sorcery_memory_cache menuselect.makeopts
make -j4
make install

# 4. Remove stale lib64 libraries (causes install warning)
rm -f /usr/lib64/libasteriskpj.so* /usr/lib64/libasteriskssl.so*
ldconfig

# 5. Rebuild custom teams modules against v22
cp /usr/src/asterisk-18.26.4/res/res_pjsip_teams_contact.c /usr/src/asterisk-22.4.1/res/
cp /usr/src/asterisk-18.26.4/res/res_pjsip_teams_secure_rx.c /usr/src/asterisk-22.4.1/res/
cd /usr/src/asterisk-22.4.1 && make -j4 res
cp res/res_pjsip_teams_contact.so /usr/lib/asterisk/modules/
cp res/res_pjsip_teams_secure_rx.so /usr/lib/asterisk/modules/

# 6. Move incompatible Asterisk 18 modules out (ABI incompatible — cause SEGV)
mkdir -p /usr/lib/asterisk/modules.disabled-18
mv /usr/lib/asterisk/modules/codec_g729a.so /usr/lib/asterisk/modules.disabled-18/ 2>/dev/null
mv /usr/lib/asterisk/modules/codec_g729a.manifest.xml /usr/lib/asterisk/modules.disabled-18/ 2>/dev/null
mv /usr/lib/asterisk/modules/codec_g729.so /usr/lib/asterisk/modules.disabled-18/ 2>/dev/null

# 7. Update res_odbc.conf — replace pooling/limit/idlecheck with max_connections
# Old:  pooling => yes / limit => 10 / idlecheck => 60000
# New:  max_connections => 10

# 8. Start
systemctl restart asterisk
```

### Verification

```bash
asterisk -rx "core show uptime"          # Should show 22.4.1
asterisk -rx "pjsip show transports"     # UDP/TCP/TLS/WSS all bound
asterisk -rx "pjsip show registrations"  # ECN trunk: Registered
asterisk -rx "pjsip show contacts"       # Teams: Avail ~190ms RTT
asterisk -rx "module show like codec_opus"    # Running
asterisk -rx "module show like res_pjsip_nat" # Running (patched)
asterisk -rx "module show like teams"         # teams_contact: Running
```

### Backed-up files

| Path | Contents |
|---|---|
| `/usr/lib/asterisk/modules.backup-18.26.4/` | All Asterisk 18 compiled modules |
| `/usr/lib/asterisk/modules.disabled-18/` | ABI-incompatible modules (g729a, g729) |
| `/usr/src/asterisk-22.4.1/res/res_pjsip_nat.c.orig` | Unpatched NAT source |
| `/usr/sbin/asterisk.18.26.4.bak` | Asterisk 18 binary |

---

## Re-running on a New Server

```bash
# Clone the repo, then:
sudo bash /var/www/html/scripts/pbx-server-optimise.sh
```

The script handles swap, kernel tuning, systemd limits, Opus build, logger config, and database backfill automatically.