The Top Right Corner

In football there are free-kicks so superbly taken that commentators will say the goalie had no chance. The ball was so perfectly arced into the top right corner that there was nothing the hapless keeper could have done to stop it. What brings me to this? A trouble-shooting scenario where the issue was so deviously constructed – albeit not on purpose – that it felt to some extent like that free kick.

Some colleagues were building out a network on a remote site where the traffic went out to the Internet through a pair of Fortigates and access to the Internet had stopped working. They rang me as I had built the firewalls. I had no remote access but there was a clued-up guy on site who ran a debug which told us the policy was blocking the traffic out. However, the policy out from the VLAN in question allowed access to ‘All’ destinations for http, https and dns but for some reason the traffic wasn’t hitting it. Access to the Internet was working from the firewall itself. There was an SD-WAN policy in place so I thought this was somehow to blame as in was the only ‘moving-part’ but various troubleshooting steps cleared this as a cause. It wasn’t a production system and it was late in the afternoon so I asked for the debugs to be sent to me wondering if we’d hit an obscure bug.

The issue came to light before the logs came across. Someone had somehow changed the ‘all’ object from 0.0.0.0/0 to something else – I can’t remember the details but the upshot was that it no longer acted as an ‘all’ object, though was still named that, so the rule we thought was allowing traffic to all destinations was only allowing it to a very specific destination and thus the rule had effectively become a deny ‘all’ despite looking very much like the opposite.

Everyday’s a school day.