Issue #284
Before working with Windows Phone and iOS, my life involved researching VoIP. That was to build a C library for voice over IP functionality for a very popular app, and that was how I got started in open source.
The library I was working with were Linphone and pjsip. I learn a lot of UDP and SIP protocol, how to build C library for consumption in iOS, Android and Windows Phone, how challenging it is to support C++ component and thread pool in Windows Phone 8, how to tweak entropy functionality in OpenSSL to make it compile in Windows Phone 8, how hard it was to debug C code with Android NDK. It was time when I needed to open Visual Studio, Xcode and Eclipse IDE at the same time, joined mailing list and followed gmane. Lots of good memories.
Today I find that those bookmarks I made are still available on Safari, so I think I should share here. I need to remove many articles because they are outdated or not available anymore. These are the resources that I actually read and used, not some random links. Hopefully you can find something useful.
This post focuses more about resources for pjsip on client and how to talk directly and with/without a proxy server.
First of all
Here are some of the articles and open sources made by me regarding VoIP, hope you find it useful
rtpproxy: I forked from http://www.rtpproxy.org/ and changed code to make it support for IP handover. It means the proxy can handle when IP changes from 3G, 4G to Wifi and to reduce chances of attacks
VoIP overview
Voice over Internet Protocol (also voice over IP, VoIP or IP telephony) is a methodology and group of technologies for the delivery of voice communications and multimedia sessions over Internet Protocol (IP) networks, such as the Internet
Voice over IP Overview: introduction to VoIP concepts, H.323 and SIP protocol
Voice over Internet Protocol the wikipedia article contains very foundation knowledge
Open Source VOIP Software: this is a must read. Lots of foundation articles about client and server functionalities, SIP, TURN, RTP, and many open sources framworks
VOIP call bandwidth: a very key factor in VoIP application is bandwidth consumption, it’s good to not going far beyond the accepted limit
Routers SIP ALG: this is the most annoying, because there is NAT and many types of NAT, also router with SIP ALG
SIP SIMPLE Client SDK: introduction to SIP core library, but it gives an overview of how
SIP
The Session Initiation Protocol (SIP) is a communications protocol for signaling and controlling multimedia communication sessions in applications of Internet telephony for voice and video calls, in private IP telephone systems, as well as in instant messaging over Internet Protocol (IP) networks.
RFC 3261: to understand SIP, we need to read its standard. I don’t know how many times I read this RFC.
OpenSIPS: OpenSIPS is a multi-functional, multi-purpose signaling SIP server
SIP protocol structure through an example: this is a must read, it shows very basic but necessary knowledge
Relation among Call, Dialog, Transaction & Message: basic concepts about call, dialog, transaction and message
microSIP: Open source portable SIP softphone for Windows based on PJSIP stack. I used to use this to test my pjsip tweaked library before building it for mobile
What is SIP: introduction to SIP written by the author of CSipSimple
SIP by Wireshack: introduction to SIP written by Wireshack. I used Wireshack a lot to intercept and debug SIP sessions
Solving the Firewall/NAT Traversal Issue of SIP: this shows how NAT can be a problem to SIP applications and how NAT traversal works
SipML5 SIP client written in Javascript
SIP Retransmissions: what and how to handle retransmission
draft-ietf-sipping-dialogusage-06: this is a draft about Multiple Dialog Usages in the Session Initiation Protocol
Creating and sending INVITE and CANCEL SIP text messages: SIP also supports sending text message, not just audio and video packages. This isa good for chat application
SIP server
- Kamailio: this is the server that I used, and it plays well with lots of standard SIP clients, including pjsip. Debugging on this server was also a fun story
Configuring NAT traversal using Kamailio 3.1 and the Rtpproxy server: I don’t know how many times I had read this post
How to set up and use SIP Server on Windows: I used this to test a working SIP server on Windows
OpenSIPS/Kamailio serving far end nat traversal: discussion about how Kamailio deals with NAT traversal
NAT Traversal Module: how NAT traversal works in Kamailio as a module
RFC
RTP, SIP clients and server need to conform to some predefined protocols to meet standard and to be able to talk with each other. You need to read RFC a lot, besides you need to read some drafts.
NAT
NAT solves the problem with lack of IP, but it causes lots of problem for SIP applications, and for me as well 😂
Network address translation: Network address translation (NAT) is a method of remapping one IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device
Configuring Port Address Translation (PAT): how to configure port forwarding
Types Of NAT Explained (Port Restricted NAT, etc): This is a must read. I didn’t expect there’s many kinds of NAT in real life, and how each type affects SIP application in its own way
One Way Audio SIP Fix: sometimes we get the problem that only 1 person can speak, this talks about why
NAT traversal for the SIP protocol: explains RTP, SIP and NAT
SIP NAT Traversal: This is a must read. How to make SIP work under NAT
NAT and Firewall Traversal with STUN / TURN / ICE: pjsip and Kamailio actually supports STUN, TURN and ICE protocol. Learn about these concepts and how to make it work
Introduction to Network Address Translation (NAT) and NAT Traversal
TCP
Learn how TCP helps SIP in initiating session and to turn in TCP mode for package sending
Transmission Control Protocol: The Transmission Control Protocol (TCP) is one of the main protocols of the Internet protocol suite. It originated in the initial network implementation in which it complemented the Internet Protocol (IP)
Datagram socket: A datagram socket is a type of network socket which provides a connectionless point for sending or receiving data packets.[2] Each packet sent or received on a datagram socket is individually addressed and routed
TCP RST packet details: learn the important of RST bit
RST packet sent from application when TCP connection not getting closed properly
Why will a TCP Server send a FIN immediately after accepting a connection?
Where do resets come from? (No, the stork does not bring them.): learn about 3 ways handshake in TCP connection
Sockets and Ports: Do not confuse between socket and port
TCP Wake-Up: Reducing Keep-Alive Traffic in Mobile IPv4 and IPsec NAT Traversal
TLS
Learn about Transport Layer Security and SSL, especially openSSL for how to secure SIP connection. The interesting thing is to read code in pjsip about how it uses openSSL to encrypt messages
Configuring TLS support in Kamailio 3.1 — Howto: learn how to enable TLS mode in Kamailio
SIP TLS: how to configure TLS in Asterisk
ICE
Learn about Interactive Connectivity Establishment, another way to workaround NAT
STUN and TURN
Learn about Session Traversal Utilities for NAT and Traversal Using Relays around NAT, another way to workaround NAT
STUN: STUN (Simple Traversal of UDP through NATs (Network Address Translation)) is a protocol for assisting devices behind a NAT firewall or router with their packet routing. RFC 5389 redefines the term STUN as ‘Session Traversal Utilities for NAT’.
ALG
Learn about [Application Layer Gateway](http://Application Layer Gateway) and how it affects your SIP application. This component knows how to deal and modify your SIP message, so it might introduce unexpected behaviours.
What is SIP ALG and why does Gradwell recommend that I turn it off?
Understanding SIP with Network Address Translation (NAT): This is a must read, a very thorough document
Voice quality
Learn about voice quality, bandwidth and fixing delay in audio
RTP, Jitter and audio quality in VoIP: learn about the important of jitter and RTP
An Adaptive Codec Switching Scheme for SIP-based VoIP: explain codec switching during call in SIP based VoIP
Echo
This is a very common problem in VoIP, sometimes we hear voice from the other and also from us. Learn how echo is made, and how to effectively do echo cancellation
Echo Cancellation: How to use Speex to cancel echo
Echo and Sidetone: A telephone is a duplex device, meaning it is both transmitting and receiving on the same pair of wires. The phone network must ensure that not too much of the caller’s voice is fed back into his or her receiver
How software echo canceller works?: I asked about how we use software to do echo cancellation
Dual Tone
Learn how to generate dual tone to make signal in telecommunication
pjsip
PJSIP is a free and open source multimedia communication library written in C language implementing standard based protocols such as SIP, SDP, RTP, STUN, TURN, and ICE. It combines signaling protocol (SIP) with rich multimedia framework and NAT traversal functionality into high level API that is portable and suitable for almost any type of systems ranging from desktops, embedded systems, to mobile handsets.
PJSUA API — High Level Softphone API: high level usage of pjsip
Stateful Operations: common functions to send request statefully
Message Creation and Stateless Operations: functions related to send and receive messages
Understanding Media Flow: this is a must read. The media layer is so important, it controls sound, codec and conference bridge.
Getting Started: Building and Using PJSIP and PJMEDIA: This article describes how to download, customize, build, and use the open source PJSIP and PJMEDIA SIP and media stack
Codec Framework: pjsip supports multiple codec
Adaptive jitter buffer: this takes sometime to understand, but it plays an important part in making pjsip work properly regarding buffer handling
PJSUA-API Accounts Management: how to register account in pjsua
Building Dynamic Link Libraries (DLL/DSO): how to build pjsip as a dynamic library
Compile time configuration: lots of configuration we can apply to pjsip
Fast Memory Pool: pjsip has its own memory pool. It’s very interesting to look at the source code and learn something new
Using SIP TCP Transport: How to enable TCP mode in SIP and to initiate SIP session
Monochannel and multichannel audio frame converter: interesting read about mono and multi channel
IOQueue: I/O Event Dispatching with Proactor Pattern: the code for this is very interesting and plays a fundamental in how pjsip handles events
DNS Asynchronous/Caching Resolution Engine: how pjsip handles DNS resolution by itself
Secure socket I/O: the code for this is important if you want to learn how to use SSL under the hood
Multi-frequency tone generator: I learn a lot how pjsip uses sin wave to generate tone
SIP SRV Server Resolution (RFC 3263 — Locating SIP Servers): learn the mechanism for how pjsip finds a particular SIP server
Exception Handling: how to do Try Catch in C
Mutex Locks Order in PJSUA-LIB: how multiple locks at each layer helps ensure correction and avoid deadlocks. I had lots of nightmare debugging deadlocks with pjsip 😱
Threading
pjsip uses Local Thread Storage which introduces very cool behaviors
Threads question: how pjlib handles thread
Using Thread Local Storage: how to use TlsAlloc and TlsFree in Windows
Example: Thread local storage in a Pthread program: how Pthread works
Thread Local Storage: learn about pj_thread
Resampling
How to work with sample rate of the media stream
Resample Port: how to perform resampling in pjmedia
Resampling Algorithm: code to perform resampling
Samples: Using Resample Port: very straightforward example to change sample rate of the media stream
Memory and Performance
Audio
How to Record Audio with pjsua: how to use pjsua to record audio.
Memory/Buffer-based Capture Port: believe me, you will jump into pjmedia_mem_capture_create a lot
File Writer (Recorder): record audio to .wav file
AMR Audio Encoding: understands AMR encoding
Audio Device API: how pjsip detects and use Audio device
Sound Device Port: Media Port Connection Abstraction to the Sound Device
Audio Manipulation Algorithms: lots of cool algorithm written in C for audio manipulation. The hardest and most imporant one is probably Adaptive jitter buffer
bad quality on iphone 2G with os 3.0: No one would use iPhone 2G now, but it’s good to be aware of older phones
getting Underflow, buf_cnt=0, will generate 1 frame continuessly: how to handle underflow in pjmedia
Measuring Sound Latency: This article describes how to measure both sound device latency and overall (end-to-end) latency of pjsua
Master/sound: How master sound works and deal with no sound on the mic input port
Video
I learn a lot regarding video capture, ffmpeg and color space, especially YUV
siphon — VIdeoSupport.wiki: How siphon deals with video before pjsip 2.0
Video Device API; PJMEDIA Video Device API is a cross-platform video API appropriate for use with VoIP applications and many other types of video streaming applications.
PJSUA-API Video: Uses video APIs in pjsua with pjsip 2.1.0
PJSIP Video User’s Guide: all you need to know about video support in pjsip
Video streams: I can’t never forget pjmedia_vid_stream_create
Video source duplicator: duplicate video data in the stream.
AVI File Player: Video and audio playback from AVI file
PJSIP Version 2.0 Release Notes: starting with 2.0, pjsip supports video. Good to read
FFmpeg-iOS-build-script: details how to build ffmpeg for iOS
CSipSimple
There are many SIP client for mobile and desktop, microSIP, Jitsi, Linphone, Doubango, … They all follow strictly SIP standard and may have their own SIP core, for example microSIP uses pjsip, Linphone uses liblinphone, …
Among that, I learn a lot from the Android client, CSipSimple, which offers very nice interface and have good functionalities. Unfortunately Google Code was closed, so I don’t know if the author has plan to do development on GitHub.
I also participated a lot on the Google forum for user and dev. Thanks for Regis, I learn a lot about open source and that made me interested in open source.
You can read What is a branded version
I don’t make any money from csipsimple at all. It’s a pure opensource and free as in speech project. I develop it on my free time and just so that it benefit users. That’s the reason why the project is released under GPL license terms. I advise you to read carefully the license (you’ll learn a lot of things on the spirit of the license and the project) : http://www.gnu.org/licenses/gpl.html To sump up, the spirit of the GPL is that users should be always allowed to see the source code of the software they use, to use it the way they want and to redistribute it.
RTP Proxy
Because of NAT or in case users want to talk via a proxy, then a RTP proxy is needed. RTPProxy follows standard and works well with Kamailio
Other related articles
multiple audio devices, multiple calls, conferencing, recording and mix all of the above
Conference bridge should transmit silence frame when level is zero
Add user defined NAT hole-punching and keep-alive mechanism to media stream
IP change
IP change during call can cause problem, such as when user goes from Wifi to 4G mode
RTP and RTCP
Learn about [Realtime transport control protocol](http://Real-time Transport Protocol) and how that works with RTP
Codec
To reduce payload size, we need to encode and decode the audio and video package. We usually use Speex and Opus. Also, it’s good to understand the .wav format
Building pjsip for Windows Phone 8
Windows Phone 8 introduces C++ component , changes in threading, VoIP and audio background mode. To do this I need to find another threadpool component and tweak openSSL a bit to make it compile on Windows Phone 8. I lost the source code so can’t upload the code to GitHub 😢. Also many links broke because Nokia was not here any more
Porting to New CPU Architecture: pjlib is the foundation of pjsip. Learn how to port it to another platform
How to implement audio streaming for VoIP calls for Windows Phone 8
Porting OpenSSL to Windows Phone 8
Firstly, learn how to compile, use OpenSSL. How to call it from pjsip, and how to make it compile in Visual Studio for Windows Phone 8. I also learn the important of Winsock, how to port a library. I struggled a lot with porting openSSL to Windows RT, then to Windows Phone 8
A lot of links were broken 😢 so I can’t paste them all here.
C and C++
Since pjsip, rtpproxy and kamailio are all C and C++ code. I needed to have a good understanding about them, especially pointer and memory handling. We also needed to learn about compile flags for debug and release builds, how to use Make, how to make static and dynamic libraries.
comp.lang.c Frequently Asked Questions: there’s lot of things about C we haven’t known about
Bit Twiddling Hacks: how to apply clever hacks with bit operators. Really really good reading here
Better types in C++11 — nullptr, enum classes (strongly typed enumerations) and cstdint