阅读webrtc源码,从examples中的peerconnection开始。版本m98
。
一、 基本流程
server端只是做了一个http server,来转发client端的消息。也就是起到了信令服务器的作用,本篇文章不在研究,感兴趣的可以学习一下用cpp搭建http server。
二、Client代码基本流程
client界面部分代码分为linux和windows两个平台,我们以linux为例进行分析。
- 首先,在main函数里调用
GtkMainWnd::Create()
创建了gtk
的窗口界面; - 然后调用
SwitchToConnectUI()
切换到连接ui界面; - 在这个界面里有个名字为
Connect
的button
,button对应的点击事件会调用OnClickedCallback
函数。
g_signal_connect(button, "clicked", G_CALLBACK(OnClickedCallback), this);
OnClickedCallback
>GtkMainWnd::OnClicked
>Conductor::StartLogin
>PeerConnectionClient::Connect
- 在
PeerConnectionClient::Connect
函数中,会设置server的ip和port,以及client自己的name,然后调用DoConnect()
函数真正的进行连接。
void PeerConnectionClient::DoConnect() {
control_socket_.reset(CreateClientSocket(server_address_.ipaddr().family()));
hanging_get_.reset(CreateClientSocket(server_address_.ipaddr().family()));
InitSocketSignals(); //设置连接的一些callback
char buffer[1024];
snprintf(buffer, sizeof(buffer), "GET /sign_in?%s HTTP/1.0\r\n\r\n",
client_name_.c_str());
onconnect_data_ = buffer;
bool ret = ConnectControlSocket(); //发送sign in到server
if (ret)
state_ = SIGNING_IN;
if (!ret) {
callback_->OnServerConnectionFailure();
}
}
- server返回的消息均在
PeerConnectionClient::OnRead
里进行处理,向server发送sign in
后会调用Conductor::OnSignedIn()
,然后切换至peer list。 - 在Peer List界面中会把其他的client全部添加到列表中。
2.1 主动发起连接端
选择列表中的client后可以触发OnRowActivatedCallback
函数,从而调用Conductor::ConnectToPeer
。
void Conductor::ConnectToPeer(int peer_id) {
RTC_DCHECK(peer_id_ == -1);
RTC_DCHECK(peer_id != -1);
if (peer_connection_.get()) {
main_wnd_->MessageBox(
"Error", "We only support connecting to one peer at a time", true);
return;
}
if (InitializePeerConnection()) { //初始化PeerConnection
peer_id_ = peer_id;
peer_connection_->CreateOffer( //生成Offer
this, webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());
} else {
main_wnd_->MessageBox("Error", "Failed to initialize PeerConnection", true);
}
}
2.1.1 初始化PeerConnection
bool Conductor::InitializePeerConnection() {
RTC_DCHECK(!peer_connection_factory_);
RTC_DCHECK(!peer_connection_);
if (!signaling_thread_.get()) {
signaling_thread_ = rtc::Thread::CreateWithSocketServer();
signaling_thread_->Start();
}
peer_connection_factory_ = webrtc::CreatePeerConnectionFactory(
nullptr /* network_thread */, nullptr /* worker_thread */,
signaling_thread_.get(), nullptr /* default_adm */,
webrtc::CreateBuiltinAudioEncoderFactory(),
webrtc::CreateBuiltinAudioDecoderFactory(),
webrtc::CreateBuiltinVideoEncoderFactory(),
webrtc::CreateBuiltinVideoDecoderFactory(), nullptr /* audio_mixer */,
nullptr /* audio_processing */);
if (!peer_connection_factory_) {
main_wnd_->MessageBox("Error", "Failed to initialize PeerConnectionFactory",
true);
DeletePeerConnection();
return false;
}
if (!CreatePeerConnection()) {
main_wnd_->MessageBox("Error", "CreatePeerConnection failed", true);
DeletePeerConnection();
}
AddTracks();
return peer_connection_ != nullptr;
}
- 首先
CreatePeerConnectionFactory
,参数中设定了Audio和Video的编解码工厂,全部使用webrtc内置的编解码器。如果想用自己的编解码器就可以从这里入手,实现自己的编解码工厂和编解码器,并且在CreatePeerConnectionFactory
中指定。 - 然后利用
peer_connection_factory_
来创建PeerConnection
。 - 最后是添加音视频
tracks
。一个stream
中可以包含音频track
和视频track
。每个track
中可以有一个source
和多个sink
。source
可以理解为音视频的生产者,sink
可以理解为音视频的消费者。
void Conductor::AddTracks() {
if (!peer_connection_->GetSenders().empty()) {
return; // Already added tracks.
}
//生成audio source和track并添加到peerconnection中
rtc::scoped_refptr<webrtc::AudioTrackInterface> audio_track(
peer_connection_factory_->CreateAudioTrack(
kAudioLabel, peer_connection_factory_->CreateAudioSource(
cricket::AudioOptions())));
auto result_or_error = peer_connection_->AddTrack(audio_track, {kStreamId});
if (!result_or_error.ok()) {
RTC_LOG(LS_ERROR) << "Failed to add audio track to PeerConnection: "
<< result_or_error.error().message();
}
//生成video source和track并添加到peerconnection中
rtc::scoped_refptr<CapturerTrackSource> video_device =
CapturerTrackSource::Create();
if (video_device) {
rtc::scoped_refptr<webrtc::VideoTrackInterface> video_track_(
peer_connection_factory_->CreateVideoTrack(kVideoLabel, video_device));
main_wnd_->StartLocalRenderer(video_track_); //把localRenderer作为sink添加到video track中,起到本地视频预览的作用
result_or_error = peer_connection_->AddTrack(video_track_, {kStreamId});
if (!result_or_error.ok()) {
RTC_LOG(LS_ERROR) << "Failed to add video track to PeerConnection: "
<< result_or_error.error().message();
}
} else {
RTC_LOG(LS_ERROR) << "OpenVideoCaptureDevice failed";
}
//切换到streaming ui
main_wnd_->SwitchToStreamingUI();
}
2.1.2 CreateOffer
初始化PeerConnection
成功以后,发起端会调用CreateOffer
函数生成Offer
。
peer_connection_->CreateOffer( //生成Offer
this, webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());
CreateOffer
函数的第一个参数是CreateSessionDescriptionObserver
。生成Offer成功会回调CreateSessionDescriptionObserver
的OnSuccess
。而Conductor
类继承于CreateSessionDescriptionObserver
,所以生成Offer成功后会回调Conductor::OnSuccess
。
void Conductor::OnSuccess(webrtc::SessionDescriptionInterface* desc) {
peer_connection_->SetLocalDescription(
DummySetSessionDescriptionObserver::Create(), desc); //设置local description
std::string sdp;
desc->ToString(&sdp); //将offer转换为字符串
// For loopback test. To save some connecting delay.
if (loopback_) {
// Replace message type from "offer" to "answer"
std::unique_ptr<webrtc::SessionDescriptionInterface> session_description =
webrtc::CreateSessionDescription(webrtc::SdpType::kAnswer, sdp);
peer_connection_->SetRemoteDescription(
DummySetSessionDescriptionObserver::Create(),
session_description.release());
return;
}
Json::StyledWriter writer;
Json::Value jmessage;
jmessage[kSessionDescriptionTypeName] =
webrtc::SdpTypeToString(desc->GetType());
jmessage[kSessionDescriptionSdpName] = sdp;
SendMessage(writer.write(jmessage)); //将offer发送给server,由server转发给peer client
}
在OnSuccess
函数中,首先设置peerconnection
的local description(这是webrtc的连接的基本流程之一),然后转换为字符串发送给对端。
2.2 被动连接端
被动端跟主动端一样,先在server中注册,然后通过PeerConnectionClient::OnHangingGetRead
读取server端的信息,如果收到主动端的信息,就会调用OnMessageFromPeer
函数。
void Conductor::OnMessageFromPeer(int peer_id, const std::string& message) {
RTC_DCHECK(peer_id_ == peer_id || peer_id_ == -1);
RTC_DCHECK(!message.empty());
if (!peer_connection_.get()) {
RTC_DCHECK(peer_id_ == -1);
peer_id_ = peer_id;
//初始化peerconnection
if (!InitializePeerConnection()) {
RTC_LOG(LS_ERROR) << "Failed to initialize our PeerConnection instance";
client_->SignOut();
return;
}
} else if (peer_id != peer_id_) {
RTC_DCHECK(peer_id_ != -1);
RTC_LOG(LS_WARNING)
<< "Received a message from unknown peer while already in a "
"conversation with a different peer.";
return;
}
Json::Reader reader;
Json::Value jmessage;
if (!reader.parse(message, jmessage)) {
RTC_LOG(LS_WARNING) << "Received unknown message. " << message;
return;
}
std::string type_str;
std::string json_object;
rtc::GetStringFromJsonObject(jmessage, kSessionDescriptionTypeName,
&type_str);
if (!type_str.empty()) {
if (type_str == "offer-loopback") {
// This is a loopback call.
// Recreate the peerconnection with DTLS disabled.
if (!ReinitializePeerConnectionForLoopback()) {
RTC_LOG(LS_ERROR) << "Failed to initialize our PeerConnection instance";
DeletePeerConnection();
client_->SignOut();
}
return;
}
absl::optional<webrtc::SdpType> type_maybe =
webrtc::SdpTypeFromString(type_str);
if (!type_maybe) {
RTC_LOG(LS_ERROR) << "Unknown SDP type: " << type_str;
return;
}
webrtc::SdpType type = *type_maybe;
std::string sdp;
if (!rtc::GetStringFromJsonObject(jmessage, kSessionDescriptionSdpName,
&sdp)) {
RTC_LOG(LS_WARNING)
<< "Can't parse received session description message.";
return;
}
webrtc::SdpParseError error;
std::unique_ptr<webrtc::SessionDescriptionInterface> session_description =
webrtc::CreateSessionDescription(type, sdp, &error); //根据对端sdp字符串生成webrtc的sdp
if (!session_description) {
RTC_LOG(LS_WARNING)
<< "Can't parse received session description message. "
"SdpParseError was: "
<< error.description;
return;
}
RTC_LOG(LS_INFO) << " Received session description :" << message;
peer_connection_->SetRemoteDescription( //设置远程sdp
DummySetSessionDescriptionObserver::Create(),
session_description.release());
if (type == webrtc::SdpType::kOffer) { //如果是offer信息的话,就生成answer并发送给对方
peer_connection_->CreateAnswer(
this, webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());
}
} else {
std::string sdp_mid;
int sdp_mlineindex = 0;
std::string sdp;
if (!rtc::GetStringFromJsonObject(jmessage, kCandidateSdpMidName,
&sdp_mid) ||
!rtc::GetIntFromJsonObject(jmessage, kCandidateSdpMlineIndexName,
&sdp_mlineindex) ||
!rtc::GetStringFromJsonObject(jmessage, kCandidateSdpName, &sdp)) {
RTC_LOG(LS_WARNING) << "Can't parse received message.";
return;
}
webrtc::SdpParseError error;
std::unique_ptr<webrtc::IceCandidateInterface> candidate( //生成candidate,如果sdp中包含由candidate信息的话,不需要这个过程
webrtc::CreateIceCandidate(sdp_mid, sdp_mlineindex, sdp, &error));
if (!candidate.get()) {
RTC_LOG(LS_WARNING) << "Can't parse received candidate message. "
"SdpParseError was: "
<< error.description;
return;
}
if (!peer_connection_->AddIceCandidate(candidate.get())) { //添加candidate
RTC_LOG(LS_WARNING) << "Failed to apply the received candidate";
return;
}
RTC_LOG(LS_INFO) << " Received candidate :" << message;
}
}
收到对端信息后,进行解析,解析到sdp信息的话,安装webrtc标准流程设置sdp。如果是被动端,就会收到主动端的offer信息,那么就需要生成answer并发送给对端。
而CreateAnswer
与CreateOffer
类似,成功后会回调OnSuccess
函数,与主动端的流程一致。
2.3 视频播放
在peerconnection
的SetRemoteDescription
信息时,如果sdp中有media,则会回调PeerConnectionObserver
的OnAddTrack
函数。Conductor
继承于PeerConnectionObserver
,也就是会回调Conductor::OnAddTrack
,从而触发Conductor::UIThreadCallback
的NEW_TRACK_ADDED
。
case NEW_TRACK_ADDED: {
auto* track = reinterpret_cast<webrtc::MediaStreamTrackInterface*>(data);
if (track->kind() == webrtc::MediaStreamTrackInterface::kVideoKind) {
auto* video_track = static_cast<webrtc::VideoTrackInterface*>(track);
main_wnd_->StartRemoteRenderer(video_track);
}
track->Release();
break;
}
在这里,会像本地预览那样,将远程video track
的sink
设置为remote_renderer_
。这样当收到对端的视频数据后,就会触发remote_renderer_
的OnFrame
函数。
void MainWnd::VideoRenderer::OnFrame(const webrtc::VideoFrame& video_frame) {
{
AutoLock<VideoRenderer> lock(this);
rtc::scoped_refptr<webrtc::I420BufferInterface> buffer(
video_frame.video_frame_buffer()->ToI420());
if (video_frame.rotation() != webrtc::kVideoRotation_0) {
buffer = webrtc::I420Buffer::Rotate(*buffer, video_frame.rotation());
}
//设置宽高
SetSize(buffer->width(), buffer->height());
RTC_DCHECK(image_.get() != NULL);
//将yuv转换为RGB
libyuv::I420ToARGB(buffer->DataY(), buffer->StrideY(), buffer->DataU(),
buffer->StrideU(), buffer->DataV(), buffer->StrideV(),
image_.get(),
bmi_.bmiHeader.biWidth * bmi_.bmiHeader.biBitCount / 8,
buffer->width(), buffer->height());
}
InvalidateRect(wnd_, NULL, TRUE);
}
在Onfame
函数中,会把接收到的yuv
数据利用libyuv
库转换为rgb
数据,并存储在image_
中。
当MainWnd触发OnPaint事件时,就可以把image_
数据渲染到窗口上了。这个在不同的平台可以用不同的方法进行渲染,本文就不再分析了。
2.4 建立连接
双方交换完成offer
和answer
以后,如果sdp
中包含了candidate
信息,则会直接按照sdp
中的candidate
信息尝试连接;如果sdp
中不包含candidate
信息,则需要手动发送candidate
到对端,流程也在OnMessageFromPeer
中。这样就建立起了连接,可以进行正常的音视频交互了。