Published with St Cloud Welding’s agreement. Technical detail is from our own engineering records.

The situation

St Cloud Welding, a manufacturer in Osceola County, had ten desk phones on a hosted phone system. The phones mostly worked, which is what made the problem so difficult to pin down.

The complaint was specific: when a call came in and the whole office was supposed to ring, only some phones rang. Which ones changed from call to call. Individually every phone was fine — you could call any extension directly and it rang. Put them all in a group and one or two would sit silent.

“Intermittent” problems like this get lived with for a long time, because nothing is broken enough to justify tearing the system apart.

Root cause one: the internet connection, not the phones

The site is served by satellite internet, which puts it behind carrier-grade address translation — the provider shares one public address across many customers and maps ports dynamically.

Phone systems dislike this intensely. A handset registers and the system records the path back to it. Satellite rotates that mapping. The recorded path goes stale, and the next call for that phone is sent to an address that no longer routes anywhere. No error, no alarm. The phone simply does not ring.

That explained the randomness perfectly: whichever handsets happened to have a stale mapping at that moment were the ones that stayed quiet.

The fix was to stop depending on the satellite path. A private encrypted tunnel already existed between the site and the phone system but was not being used for calls. We enabled and repaired a dedicated internal channel on the phone system, bound it to the tunnel, and moved every handset onto it — keeping the system address as the identity the phones authenticate with, and pointing the call path down the tunnel.

All ten extensions cut over and were verified registered with no address translation in the path. The random silent phones stopped.

Root cause two: a race when everything rings at once

Most engagements end there. This one did not.

On real inbound calls from the outside world to the largest ring groups — ten and eleven phones ringing simultaneously — the last one or two legs still occasionally failed. Not stale paths this time; the phone system itself was rejecting them at the moment of routing.

It only happened with a genuine external call. Testing internally never reproduced it, which is exactly why it had survived previous attempts to diagnose it. We had to build a way to loop a real outside call back into the system on demand before we could see it happen reliably.

The cause was a race condition: firing eleven call legs in the same instant occasionally overwhelmed the routing step. Not capacity — the system was nowhere near its limits — just simultaneity.

The fix was a one-second stagger. The first six phones ring immediately, the rest a second later. It de-bursts the fork, it is imperceptible to a caller, and it is completely reversible. Validated against real inbound calls with no failures.

The third thing, while we were in there

The shortcut keys on the handsets — the ones that show whether a colleague is on a call — had gone grey. Staff had assumed that was normal.

The new internal channel was missing the presence settings the old one had. We enabled presence management and set the correct hosts. Over a hundred subscriptions came back up and the keys lit again.

The outcome

• All ten extensions stable on a private path, none exposed to satellite address translation
• Ring groups ring every phone, verified against real inbound calls
• Presence indicators working again across the office
• No new hardware and no change of provider

Why we are telling this story

Three things were wrong, and they produced one symptom. Fixing the first would have improved matters and left the client still occasionally missing calls. Fixing the first two would have left staff with dead shortcut keys they had stopped mentioning.

“Intermittent” almost never means one thing behaving inconsistently. It usually means several things overlapping. The work is staying on it after the obvious cause is fixed and the complaints have quietened down.

Being on satellite or rural internet does not mean putting up with a phone system that half works. It means the setup has to account for it.

Talk to us

iTech Plus supports manufacturers, contractors and businesses across Central Florida — Davenport, Haines City, Kissimmee, St. Cloud, Lakeland, Orlando and the surrounding Polk and Osceola county areas. We handle phone systems, networks and the connectivity underneath them.

Call (321) 221-7117 or email info@itechplus.co.