The other day started off with several calls from our main location with complaints about printing issues. Already this Friday was going to be awful. Printer problems – the bane of everything in existence. Meanwhile one of our locations was being decommissioned.
Back to the issue at hand...
The end users were suddenly being prompted whether or not they trust the driver for the copier and if they want to install it. We use Canon for our copiers, and if you don't know, they don't pay Microsoft the fees required for them to be trusted, so this happens occasionally when drivers get updated on the print server or an update on the print server causes them to reregister. Luckily it is easy, though tedious, to solve by unmapping and remapping the copier on their workstations. Not a big deal, but not fun for first thing in the morning.
As you can tell, Friday wasn't going too well. Let's fast-forward a few hours to a little before lunch time.
The following times are rough estimates and names are made up.
A call comes into the help desk and is answered by our Tier 1, Bill.
Bill: "IT this is Bill."
Caller: "Hi Bill, this is Cathy from A Location. I can't sign into my computer."
Bill: "Does it say anything on the screen? Any sort of error?"
Cathy: "It says 'No logon servers available to service this request.'"
Bill: "Oh! That's no good. When was the last time you were able to sign in on that station?"
Cathy: "Not too long ago, maybe 10 minutes."
Oh great! The start of something more bizarre."
Bill: "Odd – Cathy, can you check the back of the PC for any sort of blinky lights? Should be green and/or yellow.
Cathy: "Okay, let me check" A moment passes. "Yes, it's blinking."
Bill: "Okay, I'm going to put you on hold for a minute and check a few things out. Before I do that, is it just you that is having this issue?"
Cathy: "Martha and James are still working, but Karen is having the same problem."
Bill: "Okay, one moment". Puts caller on hold.
At this time, Bill is explaining the situation to me and we try a few things. I attempt to remote into the location print/application server. That was successful, so I've ruled out network loss at the location side. Ran an Ookla Speedtest too for the sake of it and that was normal too.
I've omitted some back and forth between Bill, Cathy, and Karen, but we've obtained their hostnames so we can do the following troubleshooting.
- Ping timed out but returned an IP address, so the hostname does exist in DNS.
- The returned IP address is accurate for that location's subnet.
- The NAC appliance is not actively blocking those workstation IP/MAC/Hostnames.
So what could it be?
Another call comes into the help desk and another tier 1 picks up, Derek.
Derek: "Derek, IT."
Caller: "Hi Derek, this is Lisa from B Location. I can't sign into my computer."
Derek: "Does it say anything when you try to sign in when it fails?"
Lisa: "'No logon servers available' --"
Derek: "Hang on one moment, please."
Derek (to me): "Josh, Lisa from B Location isn't able to sign into her computer. It sounds similar to A Location's issue."
Really? Two locations with the same issue. How can this be?
The next call could give hint to a potential cause. But, how could it make sense? I'll give a hint. All of our locations are interconnected via an MPLS connection.
10:55 AM (slightly before Lisa called in)
My supervisor gives me a call from C Location.
Super: "Hey Josh, I just finished disconnecting the network equipment at Decommissioned Location. Should be all set for Facilities to take it all back to storage."
Me: "Great, sounds good. Hey – question for you. A Location is having this issue where two of the workstations have no network connection but everything else does. And it sounds like B Location is having the same issue."
You want another hint? I'll give you the location subnets (again, made up but get's the point across).
- A Location
- Gateway: 10.1.10.63
- Subnet Mask: 255.255.255.192
- B Location
- Gateway: 10.1.10.126
- Subnet Mask: 255.255.255.192
- C Location
- Gateway: 10.1.10.1
- Subnet Mask: 255.255.255.192
As you can tell from that, each of these locations share part of the same class C. This matters when my next point is that all small locations are statically assigned.
Remember, mid-day C Location was cut offline.
Epiphany moment. Still on the phone with supervisor.
Me: "Mike – epiphany moment. A and B are both 10.1.10.x and C is also 10.1.10.x."
Mike: "Oh, you're right. These three locations are the last to share part of this class C."
Me: "Ya, let me see if I'm right".
Me: "Bill – what's their static IP assignment? What do they have set as a gateway and subnet mask?"
Bill: "Okay, let me check."
Bill: "10.1.10.1 and 255.255.255.0"
BINGO! DING DING DING.
Me: "AHA! Have her change the gateway to 10.1.10.126 and the subnet mask to 255.255.255.192."
Bill: "She has internet now!"
Me: "And the same thing for Karen?"
Bill: "Ya, did the same for her."
Me (to Derek): "You heard us, right? Have Lisa to the same thing, except the gateway she should use is 10.1.10.210."
Derek: "Yup. Works."
Uggghhhhhh... COME ON!!!! Case closed. But really this boils down to PAYING ATTENTION TO DETAIL.
You get it now, right? Yes, whoever deployed the two PCs at A Location and the PC at B Location, DID NOT PAY ATTENTION AND ASSIGN THE CORRECT GATEWAY IP AND SUBNET MASK.
Moral of this story. IPv4 configurations need to be correct. You need to make sure that the subnet mask and gateway IP are correct for the location.
You may be asking, "Well, how did it work all this time?"
Well, because it is an MPLS and because the router at those locations were routing traffic for the same 10.1.10.x subnet, there were no noticeable effects. What was happening is that those workstations were using 10.1.10.1 as their gateway and 255.255.255.0 for the subnet mask so the machines were aware of the other devices in the 10.1.10.1-254 range and all of their traffic for those PCs were being routed through the MPLS and through a completely different location. Bring down that location and Boom no internet connectivity.
Always pay attention to detail. Don't cut corners. If it is a matter of not knowing, ask questions.