Note: This was an issue I encountered in the summer of 2012.
For a while, my Xbox could not download system updates no matter what I did with network, firewall, and Ethernet settings. Online game-play worked just fine under my "standard" everyday network configuration; but not system updates.
In order to get the Xbox to to download the system updates, I would have to plug it into the cable modem directly and bypass the whole network and router. That was a pain because all my other computers and devices were offline during the update download.
The baffling thing was that when I first got the Xbox, and for a good while thereafter, system updates worked and then one day stopped. Now I don't turn the Xbox on all the time, so I don't know what may have changed between the time it last worked and when it stopped.
Once I realized that this was a problem, Apple pushed out the newer AirPort Utility and I installed it. Then I had no firewall logging to see what was happening.
Well, today, I finally figured it all out. Since a friend on MacRumors showed me I could install the older AirPort Utility; I got logging back. I turned the Xbox on today and saw a system update was available. I opened up the AirPort Utility (5.6), got to the logging section and pressed Update on the Xbox and this is what I found:
Jun 23 09:59:39 Severity:5 Associated with station Xbox MAC Address
Jun 23 09:59:39 Severity:1 WPA handshake failed with STA Xbox MAC Address likely due to bad password from client
Jun 23 09:59:39 Severity:5 Deauthenticating with station Xbox MAC Address (reserved 2).
As it turns out, regardless of the Xbox being hooked up via Ethernet, and it's configuration settings defaulted to use Ethernet over WiFi, when a system update comes out, the Xbox tries to download it via WiFi, ignoring the existing Ethernet connection.
I had an old WiFi password stored in the Xbox from a long time ago and never updated it when I changed my WiFi password. I didn't do that because the Xbox was now using Ethernet and wasn't supposed to use WiFi. I removed the WiFi settings altogether from the Xbox and restarted it, tried the system update and lo and behold, it worked on the Ethernet without me having to plug it into the cable modem directly.
All this time I was thinking that something had changed and it was either Apple's or Microsoft's fault that my AEBS and Xbox wouldn't talk to each other nicely. I was even considering buying a new router too.
This story may be insignificant in importance, but it does highlight the need to have some form of firewall logging on any and all routers - Apple are you listening?
Download the pdf version with pictures here.