Jitsi, ostel.co and ISP censorship

Earlier last week n8fr8 suspected something changed on the ostel.co server, due to many users emailing support specifically about Jitsi connectivity to ostel.co. The common question was “why did it work a few weeks ago and now it doesn’t anymore?”

The tl;dr follows, skip to keyword CONCLUSION to hear only the punch line.

To support n8fr8’s hypothesis, there was a small change to the server but I wan’t convinced it effected anything since all my clients continued to work properly, including Jitsi. Obviously something had changed but none of us knew what it was. After some testing we discovered the problem was related to insecure connections from Jitsi to UDP port 5060 on ostel.co. Secure connections (on TCP port 5061) continued to work as expected.

To make matters more confusing, I could register and make calls with two different clients (CSipSimple and Linphone) on the same network (my home ISP, Verizon FiOS) using an insecure connection to ostel.co on UDP port 5060.

At this point I was like WTF?

I went back to the server, diffed all the configs, checked server versions, connected with every client I could find that would run on any of my computers. The only change was a Kamailio upgrade from 4.0.1 to 4.0.2. A minor point release. The problem with Jitsi remained. What could the server be doing to this poor client?

I did a packet trace on the ostel.co server’s public network interface, filtered to dump packets only on UDP port 5060 that match my SIP username. I opened Jitsi and things got interesting. For the curious, here’s the utility and options I used. If you are new to operating a SIP network, ngrep is an excellent tool for debugging.

ngrep -d eth0 -t -p -W byline foo port 5060

I’ll include an excerpt (I’ve included only the relevant headers for this issue) of the initial request from Jitsi. IP addresses and usernames have been changed to protect the innocent.

U 2013/07/19 22:17:34.920749 0.0.0.0:5060 -> 66.151.32.200:5060
REGISTER sip:ostel.co SIP/2.0.
CSeq: 1 REGISTER.
From: "foo" <sip:foo@ostel.co>;tag=1eb3467e.
To: "foo" <sip:foo@ostel.co>.
Via: SIP/2.0/UDP 0.0.0.0:49152;branch=z9hG4bK-393535-2269e43afef0b312554eb419a8d0540e.
User-Agent: Jitsi2.3.4752Linux.
Contact: "foo" <sip:foo@0.0.0.0:49152;transport=udp;registering_acc=ostel_co>;expires=600.
.

#
U 2013/07/19 22:17:34.921155 66.151.32.200:5060 -> 0.0.0.0:5060
SIP/2.0 401 Unauthorized.
CSeq: 1 REGISTER.
From: “foo” <sip:foo@ostel.co>;tag=1eb3467e.
To: “foo” <sip:foo@ostel.co>;tag=e01f0de2cdfebbeefc5ff0c8eabbb8b3.2f1f.
Via: SIP/2.0/UDP 0.0.0.0:49152;branch=z9hG4bK-393535-2269e43afef0b312554eb419a8d0540e;rport=5060.
WWW-Authenticate: Digest realm=”ostel.co”, nonce=”Uen0alHp8z4d6ePDl83RtMwARltAxzQu”, qop=”auth”.
Server: kamailio (4.0.2 (x86_64/linux)).

If you read the response, you’ll see Kamailio sent 401 Unauthorized. This is normal for SIP authentication. A second client request should follow it, which should contain an Authorization header with an md5 and a nonce. When Kamailio receives this request, checks the auth database and sends a 200 OK response, the client is authenticated.

The SIP dialog looks good but Jitsi continues not to register. The dialog flow is cut off after the 401 Unauthorized response. It’s almost like something has blocked the response to the client.

Since I could register Linphone using the same account, I did the same trace for that client. Here’s the excerpt.

U 2013/07/19 22:33:18.372770 0.0.0.0:42680 -> 66.151.32.200:5060
REGISTER sip:ostel.co SIP/2.0.
Via: SIP/2.0/UDP 0.0.0.0:49153;rport;branch=z9hG4bK359459505.
From: <sip:foo@ostel.co>;tag=142131416.
To: <sip:foo@ostel.co>.
CSeq: 3 REGISTER.
Contact: <sip:foo@0.0.0.0:49153;line=65da8bffcabe8c4>.
User-Agent: LinphoneAndroid/2.1.2-1-g23b7fc0 (eXosip2/3.6.0).
.

#
U 2013/07/19 22:33:18.373112 66.151.32.200:5060 -> 0.0.0.0:42680
SIP/2.0 401 Unauthorized.
Via: SIP/2.0/UDP 0.0.0.0:49153;rport=42680;branch=z9hG4bK359459505.
From: <sip:foo@ostel.co>;tag=142131416.
To: <sip:foo@ostel.co>;tag=e01f0de2cdfebbeefc5ff0c8eabbb8b3.4065.
CSeq: 3 REGISTER.
WWW-Authenticate: Digest realm=”ostel.co”, nonce=”Uen4GlHp9u4FwHNY/uE1iQQNCfGHJiob”, qop=”auth”.
Server: kamailio (4.0.2 (x86_64/linux)).

This 401 Unauthorized response was received by the client and the follow up request with the Authorization header was sent with the correct digest. Linphone registered. I made a call. Everything worked fine. Indeed WTF?

I stared at these traces for a while to get a clue. Look again at the first line of the request from Jitsi. You’ll see a timestamp followed by two IP:port pairs. Notice the port on the first IP is 5060 and the port on the second IP is also 5060. This means that the source port used by Jitsi on my home network is UDP port 5060. In order for a response to come back to Jitsi, it must enter my network on the same port it exited. Now read the top line of the response from Kamailio. Indeed, the server sent the response to UDP port 5060.

Now look at the same flow for Linphone. There is a very different source port in that dialog. In this case, Kamailio sent the response to UDP port 42680 and Linphone received it. Also notice the IP address used by Kamailio as the destination of the response is the same one in the dialog from Jitsi.

The question remained, why can’t Jitsi get the same kind of SIP response on UDP port 5060? Why is Jitsi using a single source port for outgoing traffic anyway? That value can be dynamic. I configured Jitsi to use a different port for insecure SIP. It has an advanced configuration for SIP with the key “SIP client port”. I set this to 5062 (5061 is conventionally used for secure SIP traffic so I incremented by 2) and tried to register again.

SUCCESSSSSSSSSSSS!

To be thorough, I changed Jitsi’s SIP port again to a 5 digit number I randomly typed on my keyboard without looking.

SUCCESSSSSSSSSSSS!

So if Jitsi can register to Kamailio on any port other than UDP port 5060, WTF is going on? I had a suspicion. I tried one more test before I called it. I configured Jitsi to connect on TCP port 5060. It registered successfully. Now I know what’s going on. I have a sad 🙁

CONCLUSION

My ISP, Verizon FiOS, has a firewall running somewhere upstream (it could be on the router they provided, I haven’t checked yet) that blocks incoming UDP traffic to port 5060. This probably falls under their TOS section which forbids “running servers” since Verizon provides voice services for an additional fee on top of data service, despite both running over the same fiber connection to my house. It seems like Verizon doesn’t want their data-only customers to get in the way of that sweet cheddar delivery each month in exchange for “phone service”.

This sucks on two levels.

LEVEL 1

Why is my ISP censoring my incoming traffic when I have 5 mbps of incoming bandwidth? I assume the answer is “because they can.” *desolate frowny face*

LEVEL 2

Why doesn’t Jitsi use a dynamic source port for SIP requests? I assume the answer is “Jitsi is open source, why don’t I change this and send a patch upstream?”

Both levels are formidable challenges to overcome. Convincing Verizon to play nice on the Internet feels like a vanity project. I’m writing that off. To make a change to the SIP stack in Jitsi is well within the area of the GP team’s expertise, myself included but it’s not a trivial undertaking. Since this is a default configuration change there is probably a reason upstream devs made this choice so in addition to the programming work there’s the work to convince the developers this would be a change worth a new release.

Since this is specific to Jitsi, I’m going to follow up with the developers and see if I missed anything. Stay tuned for part two.

Thanks for listening. Stay safe!

12 comments for “Jitsi, ostel.co and ISP censorship

  1. Adam Mackler
    2013/07/28 at 9:48 am

    This is very important information. I’m annoyed it’s buried at the end of the post instead of listed prominently on the ostel.co FAQ where I would have noticed it when I was installing jitsi instead of after hours of banging my head against this problem.

    I’m sure I’m not alone.

  2. 2013/07/30 at 5:08 pm

    Hello,

    Did you ever report the issue to the Jitsi devs?

    I think it is indeed a bug in Jitsi. A quick workaround is to edit your ~/.jitsi/sip-communicator.properties file and remove lines such as:
    net.java.sip.communicator.SIP_PREFERRED_CLEAR_PORT=5060
    net.java.sip.communicator.SIP_PREFERRED_SECURE_PORT=5061
    This should reset Jitsi to its default behaviour of not binding on a specific source port.

    A discussion about this would be most welcome on the dev@jitsi.org mailing list!

    Boris

  3. Ali Veli
    2013/07/31 at 5:47 am

    Is it possible to share your kamailio+freeswitch arch. picture and installation procedure for them?

  4. lee
    2013/07/31 at 3:18 pm

    I’m developing a Chef cookbook for the full stack right now in this branch

    https://github.com/guardianproject/chef-twelvetone/tree/kamailio

    It’s not complete as of this comment. If you would like to help dev/debug, that’d be great.

    • Muz
      2013/10/01 at 12:46 am

      I have been following this cookbook since a year now and have attempted to install my own SIP server using this cookbook but have not succeeded. I wish to help. Please let me know the status and how can I help.

  5. 2013/08/18 at 8:15 am

    Your ISP is correctly protecting you agains SIP attacks, scanners etc.
    Additionally, if you have the expertise to run secure services on that often attacked port, you can ask your isp to open it for you.

  6. lee
    2013/08/19 at 12:06 pm

    This story is about at a home ISP, not a larger tier hosting ISP. For example, Verizon FiOS does not provide any networking support, so there’s no one available to unblock UDP port 5060. The blocking I discovered happens upstream from the home router.

  7. Travis
    2013/09/13 at 10:36 pm

    I can confirm this appears to also be the case on AT&T Uverse. I could not connect to ostel.co with Jitsi but changing the source port allowed the communication and I was able to register and make calls. I haven’t checked ngrep to verify the exact same packet mechanics are occurring but I can assume this to be a similar situation with AT&T.

  8. Jonás
    2013/09/21 at 10:14 am

    Same problem here with jitsi and Ostel, on an ISP in Poland (UPC, although they serve more countries). I wish this post got more visibility on Ostel website.

    Thanks for the effort anyway, you helped yet another user 🙂

  9. $
    2013/11/29 at 9:42 am

    some have alledged fios default isp hardware is “SIP hostile”. Are those who experience this issue either

    ° using westell router
    or
    ° have SIP ALG from other than cisco (not-linksys) or juniper networks

    in certain fios westell hardware sip alg cannot be disabled (even if gui elements exist) and even if config file manually edited and imported

    verizon wireless (3g and lte) are sip-hostel in another way upstream for MANY customers (not including lease inanities)

    using other than standard non-encrypted sip ports sometimes useful

    time warner (road runner) in some areas is sip hostile upstream requiring itsp rtp proxying magik

  10. redjoker
    2015/08/03 at 11:18 pm

    Not 100% relates, but Verizon also seems to block guardianproject.info F-Droid repos. If I switch to T-Mobile data, no problem, if I turn off Guardian Project repos and use my FiOS WiFi, no issue. I can only conclude FiOS is interfering with downloading from guardianproject.info. I also can’t access the site from FiOS, but can from T-Mobile on any browser.

    • Hans-Christoph Steiner
      2015/08/05 at 6:44 am

      To your ISP, accessing https://guardianproject.info and using our FDroid repo will look like the same thing. It is very unlikely that they are doing detailed deep packet inspection just to filter out the traffic related to FDroid. https://guardianproject.info/fdroid is hosted like any other static files on our website. The problem most likely lies elsewhere.

Leave a Reply

Your email address will not be published. Required fields are marked *