The Symptom

This morning I spent over an hour trying to publish a new update of Store Locator Plus to the public WordPress extensions directory.   It failed, multiple times.  I assumed it was something wrong with our repository so I decided to move on to something else until the remote server was fixed.

My next task was to get the WordPress language translation tools into my dev kit so we can start providing better international support.  I decided to fetch the latest language dev kit with subversion via the standard checkout:

svn co http://svn.automattic.com/wordpress-i18n/tools/trunk/

Here are a couple of the many failure messages I received back after about 10 minutes:

svn: PROPFIND of '/wordpress-i18n/tools/trunk': could not connect to server (http://svn.automattic.com)

svn: PROPFIND of '/wordpress-i18n/!svn/vcc/default': could not connect to server (http://svn.automattic.com)

My first assumption was that my virtual machine was having network issues.  I reset the network, then shut down & restarted the virtual machine.  No luck.   I then tried directly from the host.  Again no luck.  I decided to go to the office and try it from there.  Maybe my modem or router at the house was causing issues.

A Clue

I get to the office and try again.  Same problems.    Odd, more Googling was in my future.   After reading a lot of articles about proxy servers with svn (I don’t use a proxy) and doing all the “svn tricks” I know and that I could dig up online, I stumbled across an interesting post at Stack Overflow.  This is what caught my attention:

Update

I had a co-worker test this out on his home connection — he uses Comcast as well. He got the same error as I did. So it appears to be some Comcast-related issue specific to the WordPress svn repository. I was able to checkout other public repositories via http (e.g. from Google Code) just fine.

Huh, that’s interesting.  I too could use SVN with a variety of other services.  I also was using Comcast at the home office and on one-half of the network at the corporate office.   So I decided to try a couple of things.

The Test

First, shut off the Comcast connection at the office and force my system to connect via the T1. Guess, what?  It worked.  The repo was cloned immediately.

Interesting.   Second test, log in to our server in Michigan on multiple backbones, NONE of which are on Comcast.   Hey, look at that… it worked immediately as well.

Back to the office services.  Turn off the T1, turn on just Comcast.  Instant fail.   Well not instant, it waits for about 5 minutes then fails.

Bandwidth Caps

In addition to the failure to pull subversion content directly from the WordPress IP addresses, we have also found several other interesting things about Comcast Business Class Internet.   Comcast is billing us for 50Mbps/10Mbps speeds at both my home office and our corporate office locations.  We have NEVER been able to get anything close to that at either location.  Our download speeds always seem to max out around 20Mbps/4Mbps at home and 26Mbps/6Mbps at the office.

Today, in an effort to understand what may be going on, we ran ShaperProbe from GA Tech.   It turns out that at my home office we are dropping so many upload packets that many tools, including ShaperProbe fail.   We also learned the Comcast is THROTTLING the incoming bandwidth at the home office to 17Mbps, less than HALF the 50Mbps promised and paid for.   This is one method used to ensure all users have some bandwidth when they oversell a neighborhood.   Ouch.

Comcast Traffic Shaping Test

After the “network improvement” work that Comcast did this week our Corporate line is now crawling at 1/10th the advertised download rate.    We are able to receive our packets from WordPress, but now we can’t get more than a handful of simple transfers going at the same time.

Comcast Speed Test December 2011

Comcast Fails

I am still doing research on this issue and will post updates here.  However it is very obvious from the initial tests that Comcast is doing some sort of traffic shaping or other network manipulation on their business class services and it is “breaking the Internet”.

I’ll try contacting them, but I am 99.999% certain that whomever I get ahold of will be clueless.  They usually are.    In fact I bet the first thing they ask me to do is reboot my computer, then turn off the modem.  Then they’ll bounce the modem remotely.

We have called Comcast Business services and we are quite shocked to have reached Terry at Business Class Services.  She actually emailed us and is escalating the problem to a higher level tech and is going to chase this down for us today, on a Saturday of all days.   Wow.  That was surprising!  Now lets all pray for some results as doing this proxy thing is a pain!

Reaching Comcast

http://business.comcast.com/smb/contact

Business Customer Service:  (800) 391-3000

Residential Customer Service: (800) 266-2278

In the meantime if you are having the same issues with Comcast please share.   Especially if you found a viable workaround to the issue.

Tracking The Issue

Here are some related articles we’ve dug up about this issue:

• Our YouTube video showing the Comcast failure.
• WordPress Forums “subversion error copying my plugin…”
• WordPress Forums “could not connect to server”
• @nacin is Tweeting people as he digs into this.
• The Stack Overflow post that gave us “The Clue”
• An article about Net Nuetrality and Comcast blocking/filtering traffic.

Update

Similar Articles

Related Posts & Comments:

1. Charleston High Speed Internet – Part 2.5a
2. Charleston, SC High Speed Internet ISPs
3. Ubuntu – No Audio When Playing Videos
4. HTTP Errors When Uploading/Connecting in WordPress

4 Comments   |   Posted in blog,Information Technology,Networking Utilities,System Administration,Tips & Tricks by lcleveland on December 17, 2011

3 Comments for this entry

• Steve December 18th, 2011 on 10:07 PM

  Seems to be fixed now. Well, at least for me.

  Reply
• lcleveland December 19th, 2011 on 9:43 AM

  Still down here at corporate & at the home office.

  Comcast is “looking into it” but “can’t find anything wrong with the routing”. We know it is not a routing issue, we can access the servers fine. The problem is certain packets coming back are never getting to us. It is a packet inspection & filtering issue. We’re going to put our packet analysis tool online & post results. Unfortunately it is a TON of data, thousands of lines of data for a few seconds of listening, that we need to sort through.

  Reply
• Simon January 18th, 2012 on 9:01 PM

  I’m up here in Vancouver BC and this is really starting to annoy me. I can’t do an svn update!

  I am powerless being in Canada though.

  Reply

1 Trackback or Pingback for this entry

• Committing to WordPress svn via proxy | James Fishwick, January 5th, 2012 on 3:53 PM

  [...] After I bit of research, I stumbled across this link: http://www.cybersprocket.com/2011/tips-tricks/is-comcast-playing-big-brother-with-your-internet/. [...]