game server connection issues

[Lexington]

Hi there, I've created an account on this forum because I seem to have exhausted my internet research skills for this problem, and hope one of you guys can help me out.

My flatmate recently changed our router to a TP-Link WR1043ND running Gargoyle. When this happened I became unable to play Dota 2. I can connect to Steam, open the game, and enter matchmaking just fine, but when the game starts and my computer tries to connect to the server I just get:
"Server connection did not have the correct challenge, ignoring."
over and over again.

a dev on this forum suggests that means something is messing with the packets.

Google doesn't seem to bring up many other people having the same problem with this router -I'm guessing that's because of the relatively small Gargoyle userbase.

Things I have tried:
-Setting up static IP and port forwarding (although devs state it's not needed at all for Dota 2)
-Upgrading router firmware (from 1.3.x to 1.5.9)
-Connecting through PuTTY and disabling the router firewall as suggested here

Not sure what to try next.
The router is currently at the default settings, aside from having a wireless network and setup of PPPoE. No QoS or restrictions or anything.

Any help appreciated

[pbix]

I don't know what the source of you problem might be but I might check if you are running out of connections or CPU time. Both of these are shown on the opening status screen.

Either of these might cause your router to drop a packet or two which might annoy your game.

You might also try lowering your MTU size a little from 1500 to say 1492 or 1454.

What is your WAN upload/download speed?

I would also advise you learn how to use Wireshark. Packet inspection is likely in your future if you want to solve this.

[Lexington]

Thanks for your suggestions, pbix!

I checked the router status page last night while trying to connect to a game.

Connections staying at around 200/4096
CPU load about 0.19 - 0.31
memory usage around 50% (14.3/28.6 MB)

speedtest.net says 10Mbps download, 0.78Mbps upload.

The thing is, the game works fine if I just swap out this router for the old one (a netcomm something-or-other with standard firmware).
We mainly switched to the TP-Link because it gave much better coverage in the far corners of the house, for various wireless appliances.

set MTU to 1492: no change.

Re-flashed down to 1.4.7 (1.5.9 was hanging when I tried to save settings): no change.

I installed wireshark and ran a capture while I tried to connect. What sort of stuff should I be looking for? I'm pretty new to this.

[pbix]

Using Wireshark you would look for any difference in the packets sent and received from the server reporting the error using the two routers.

If you can find a difference then perhaps something could be done.

The things to look at are the length, order and contents of the packets immediately preceding the error message.

[Lexington]

okay so what i have is this
(screenshot)

I've filtered it to only show packets to/from the game server that the console says it is connecting to. The one on the left is with the Gargoyle router, on the right is with the stock router.

They diverge after the third packet. So I think this is where I should start to look for a problem. But I'm not sure which part of the packet content should be expected to be different between the two attempts, and which part should be expected to be identical.

[Lexington]

So I dumped the data from each of those two packets into excel to compare them (if there is a better way, let me know, I'm pretty new at this). Results here. On the left is the packet from the failed connection, and bytes that are identical to the successful connection are green. They are identical except for a few bytes at the beginning and near the end of the packet.
But I don't really know what to do with this information now since I don't know what they should look like.

[pbix]

If the lengths, content and type of packets between your computer and the router look the same you might look between the router and the modem.

For that you will need a hub (not a switch) in order to intercept the packets on a laptop running Wireshark. If Gargoyle is changing the packets in any way you will see it there.

I did not look at the results you posted and have no comment about that. I am just trying to give you some hints on were to look for a problem.