Live555用做RTSPClient时,利用RTP时间戳进行音视频同步的解决方案(必须有RTCP支持才可行)

先看来自Live555官网的2个常见问题
问题1:Why do most RTP sessions use separate streams for audio and video? How can a receiving client synchronize these streams?
回答:Sending audio and video in separate RTP streams provides a great deal of flexibility. For example, this makes it possible for a player to receive only the audio stream, but not video (or vice-versa). It would even be possible to have one computer receive and play audio, and a separate computer receive and play video.

These audio and video streams are synchronized using RTCP “Sender Report” (SR) packets – which map each stream’s RTP timestamp to ‘wall clock’ (NTP) time. For more information, see the IETF’s RTP/RTCP specification.

Receivers can then use this mapping to synchronize the incoming RTP streams. The LIVE555 Streaming Media code does this automatically: For subclasses of “RTPSource”, the “presentationTime” parameter that’s passed to the ‘afterGettingFunc’ of “getNextFrame()” (see “liveMedia/include/FramedSource.hh”) will be an accurate, time-synchronized time. (For this to work, you need to have also created a “RTCPInstance” for each RTP source.)

For example, if you use “openRTSP” to receive RTSP/RTP streams, then the contents of each RTP stream (audio and video) are written into separate files. This is done using the “FileSink” class. If you look at the “FileSink::afterGettingFrame()” member function, you’ll notice that there’s a “presentationTime” parameter for each incoming frame. Some other receiver could use the “presentationTime” parameter to synchronize audio and video.

问题2:But I notice that there’s an abrupt change in a stream’s presentation times after the first RTCP “SR” packet has been received. Is this a bug?
回答:No, this is normal, and expected; there’s no bug here. This happens because the first few presentation times – before RTCP synchronization occurs – are just ‘guesses’ made by the receiving code (based on the receiver’s ‘wall clock’ and the RTP timestamp). However, once RTCP synchronization occurs, all subsequent presentation times will be accurate.

This means is that a receiver should be prepared for the fact that the first few presentation times (until RTCP synchronization starts) will not be accurate. The code, however, can check this by calling “RTPSource:: hasBeenSynchronizedUsingRTCP()”. If this returns False, then the presentation times are not accurate, and should not be used for synchronization. However, once the call to returns True, then the presentation times (from then on) will be accurate.

我的心得:
1. Live555中关于写文件时,需要先同步音视频流的例子见Live555的源码QuickTimeFileSink.cpp:

void QuickTimeFileSink
::afterGettingFrame(void* clientData, unsigned packetDataSize,
            unsigned numTruncatedBytes,
            struct timeval presentationTime,
            unsigned /*durationInMicroseconds*/) {
  SubsessionIOState* ioState = (SubsessionIOState*)clientData;
  if (!ioState->syncOK(presentationTime)) {
    // Ignore this data:音视频还未同步,忽略这些数据包,syncOK中调用了hasBeenSynchronizedUsingRTCP()
    ioState->fOurSink.continuePlaying();
    return;
  }
  ...
  ioState->afterGettingFrame(packetDataSize, presentationTime);
}

2.1 通过Live555源码搜索,查找RTPSource同步标记(fCurPacketHasBeenSynchronizedUsingRTCP)设置的逻辑:
—- hasBeenSynchronizedUsingRTCP Matches (17 in 9 files) —-
MediaSession.cpp (livemedia): if (!rtpSource()->hasBeenSynchronizedUsingRTCP()) {
MultiFramedRTPSource.cpp (livemedia): fPresentationTime, fCurPacketHasBeenSynchronizedUsingRTCP,
RTPSource.cpp (livemedia): fCurPacketHasBeenSynchronizedUsingRTCP(False), fLastReceivedSSRC(0),
RTPSource.hh (livemedia\include): virtual Boolean hasBeenSynchronizedUsingRTCP();
RTPSource.hh (livemedia\include): Boolean fCurPacketHasBeenSynchronizedUsingRTCP;
testRTSPClient.cpp: if (fSubsession.rtpSource() != NULL && !fSubsession.rtpSource()->hasBeenSynchronizedUsingRTCP()) {
—- fHasBeenSyncedUsingRTCP Matches (3 in 2 files) —-
MultiFramedRTPSource.cpp (livemedia): fHasBeenSyncedUsingRTCP = hasBeenSyncedUsingRTCP;
MultiFramedRTPSource.cpp (livemedia): hasBeenSyncedUsingRTCP = fHasBeenSyncedUsingRTCP;
MultiFramedRTPSource.hh (livemedia\include): Boolean fHasBeenSyncedUsingRTCP;
—- assignMiscParams Matches (3 in 2 files) —-
MultiFramedRTPSource.cpp (livemedia): bPacket->assignMiscParams(rtpSeqNo, rtpTimestamp, presentationTime,
—- hasBeenSyncedUsingRTCP Matches (18 in 4 files) —-
MultiFramedRTPSource.cpp (livemedia): fHasBeenSyncedUsingRTCP = hasBeenSyncedUsingRTCP;
RTPSource.cpp (livemedia): resultHasBeenSyncedUsingRTCP = fHasBeenSynchronized;

2.2 真正进行RTCP时间同步标记判断和保存的地方,最终会通过bPacket->assignMiscParams函数,赋值给RTPSource::fCurPacketHasBeenSynchronizedUsingRTCP变量:

void MultiFramedRTPSource::networkReadHandler1()
{
...
    struct timeval presentationTime; // computed by:
    Boolean hasBeenSyncedUsingRTCP; // computed by:
    receptionStatsDB()
      .noteIncomingPacket(rtpSSRC, rtpSeqNo, rtpTimestamp,
              timestampFrequency(),
              usableInJitterCalculation, presentationTime,
              hasBeenSyncedUsingRTCP, bPacket->dataSize());

    // Fill in the rest of the packet descriptor, and store it:
    struct timeval timeNow;
    gettimeofday(&timeNow, NULL);
    bPacket->assignMiscParams(rtpSeqNo, rtpTimestamp, presentationTime,
                  hasBeenSyncedUsingRTCP, rtpMarkerBit,
                  timeNow);
...
}

3.结论:参考Live55源码中的QuickTimeFileSink类中关于音视频同步逻辑相关的部分,即可在自己的FileSink或者MemorySink中实现音视频同步逻辑。

其他:研读源码,发现MediaSubsession::getNormalPlayTime函数可以获取当前rtp包的ntp时间,值得一试:
—- curPacketRTPTimestamp( Matches (3 in 2 files) —-
MediaSession.cpp (livemedia): u_int32_t timestampOffset = rtpSource()->curPacketRTPTimestamp() – rtpInfo.timestamp;
MediaSession.cpp (livemedia): u_int32_t timestampOffset = rtpSource()->curPacketRTPTimestamp() – rtpInfo.timestamp;
RTPSource.hh (livemedia\include): u_int32_t curPacketRTPTimestamp() const { return fCurPacketRTPTimestamp; }
—- getNormalPlayTime Matches (4 in 3 files) —-
MediaSession.cpp (livemedia):double MediaSubsession::getNormalPlayTime(struct timeval const& presentationTime) {
MediaSession.hh (livemedia\include): double getNormalPlayTime(struct timeval const& presentationTime);
MediaSession.hh (livemedia\include): double fNPT_PTS_Offset; // set by “getNormalPlayTime()”; add this to a PTS to get NPT
testRTSPClient.cpp: envir() << “\tNPT: ” << fSubsession.getNormalPlayTime(presentationTime);

共 1 个评论(Comment)

  1. Uranus Zhou says:

    你好,我在用live555做MPEG-2 PS视频回放时发现正常播放VLC的进度条是好的,
    但是seek拖动到某个位置之后,VLC的进度条还是原来的位置,而且实际视频也是已经跳到正确的时间了,
    VLC日志里会显示:
    live555 debug: seek start: 39.621000 stop:79.147000
    main debug: Stream buffering done (26918 ms in 13 ms)
    main warning: picture is too late to be displayed (missing 26121 ms)
    这个丢失的时间就不对了,我看VLC输出里也没有
    live555 debug: tk->rtpSource->hasBeenSynchronizedUsingRTCP()
    main error: ES_OUT_RESET_PCR called
    这种输出(其它格式视频回放时有)

    请问可能是什么原因导致的?谢谢

回复

你的邮件地址不会被公开(Your email address will not be published.) Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>