探索iOS自定义ijkplayer画中画播放

iOS提供AVPictureInPictureController用于画中画播放控制，但是只能绑定AVPlayer使用，对于开发者来说挺头痛的。在iOS 15.0后呼之欲出，支持SampleBufferDisplayLayer自定义数据源显示图层，意味着我们可以自定义第三方播放器实现画中画。以ijkplayer为例，让我们一起探索如何支持画中画播放。

一、判断画中画支持

二、后台播放配置

1、配置AudioSession

2、配置后台模式

三、画中画生命周期

1、从全屏进入画中画

2、退出画中画回到全屏

3、生命周期

四、画中画创建流程

1、初始化

2、提供start和stop方法

3、实现画中画代理方法

4、更新缓冲区图层

5、更新播放状态

6、避坑指南

一、判断画中画支持

我们可以通过isPictureInPictureSupported() 判断当前设备是否支持画中画。

二、后台播放配置

1、配置AudioSession

在application创建时，配置AudioSession为playback模式，代码如下：

func application(_ application: UIApplication,
                 didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
    
    let audioSession = AVAudioSession.sharedInstance()
     do {
        try audioSession.setCategory(.playback)
     } catch {
         print("Setting category to AVAudioSessionCategoryPlayback failed.")
     }

    return true
}

2、配置后台模式

勾选“Audio, Airplay, Picture in Picture”选项，如下图所示：

三、画中画生命周期

1、从全屏进入画中画

启动画中画时，首先执行willStartPictureInPicture()方法，此时动画还未开始。等到初始化完毕，执行didStartPictureInPicture()方法，完成从全屏到画中画的切换，进入后台状态。

2、退出画中画回到全屏

退出画中画时，从后台状态恢复app的UI，执行willStopPictureInPicture()方法。最终完成退出，执行didStopPictureInPicture()方法，回到全屏播放。

3、生命周期

画中画完整生命周期包括：从全屏到画中画，最终退出画中画回到全屏。即上面两个生命周期的结合，如下图所示：

四、画中画创建流程

1、初始化

初始化AVSampleBufferDisplayLayer，包括设置frame、opaque、position、videoGravity、controlTimebase。示例代码如下：

- (AVSampleBufferDisplayLayer *) createDisplayLayer
{
    AVSampleBufferDisplayLayer *layer;
    layer = [[AVSampleBufferDisplayLayer alloc] init];
    layer.frame = self.bounds;
    layer.opaque = YES;
    layer.position = CGPointMake(CGRectGetMidX(self.bounds), CGRectGetMidY(self.bounds));
    layer.videoGravity = AVLayerVideoGravityResizeAspect;
    
    CMTimebaseRef controlTimebase;
    CMTimebaseCreateWithSourceClock( CFAllocatorGetDefault(), CMClockGetHostTimeClock(), &controlTimebase);
    layer.controlTimebase = controlTimebase;
    
    return layer;
}

使用AVSampleBufferDisplayLayer来初始化ContentSource，前面先判断设备是否支持画中画。示例代码如下：

- (BOOL)initPictureInPicture:(AVSampleBufferDisplayLayer *)displayLayer {
    if ([AVPictureInPictureController isPictureInPictureSupported] == NO) {
        NSLog(@"Sorry, don't support PictureInPicture mode!");
        return NO;
    }
    
    if (@available(iOS 15.0, *)) {
        _displayLayer  = displayLayer;
        _contentSource = [[AVPictureInPictureControllerContentSource alloc]
                            initWithSampleBufferDisplayLayer:displayLayer playbackDelegate:self];
        
        return YES;
    } else {
        return NO;
    }
}

2、提供start和stop方法

开始画中画，包括初始化AVPictureInPictureController、设置delegate代理、设置播放速率、启动。。示例代码如下：

- (void)startPictureInPicture {
    if (@available(iOS 15.0, *)) {
        if (_pipController != nil) {
            return;
        }
        _pipController = [[AVPictureInPictureController alloc] initWithContentSource:_contentSource];
        _pipController.delegate = self;
        CMTimebaseSetRate(_displayLayer.controlTimebase, 1);
        CMTimebaseSetTime(_displayLayer.controlTimebase, CMTimeMake([_delegate getCurrentTime], 1));
        // 延时启动，否则可能会启动失败
        dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(100 * NSEC_PER_MSEC)), dispatch_get_main_queue(), ^{
            if ([self->_pipController isPictureInPicturePossible]) {
                [self->_pipController startPictureInPicture];
                NSLog(@"startPictureInPicture...");
            }
        });
    }
}

结束画中画，示例代码如下：

- (void)stopPictureInPicture {
    if (_pipController != nil && [_pipController isPictureInPictureActive]) {
        [_pipController stopPictureInPicture];
    }
}

3、实现画中画代理方法

实现AVPictureInPictureControllerDelegate代理方法如下：

- (void)pictureInPictureControllerWillStartPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
    NSLog(@"pictureInPictureControllerWillStart...");
}

- (void)pictureInPictureControllerDidStartPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
    // 回调画中画开始
    [_delegate onStartPictureInPicture:nil];
}

- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController
                                    failedToStartPictureInPictureWithError:(NSError *)error {
    // 画中画开始失败
    [_delegate onStartPictureInPicture:error];
}

- (void)pictureInPictureControllerWillStopPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
    NSLog(@"pictureInPictureControllerWillStop...");
}

- (void)pictureInPictureControllerDidStopPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
    [self stopPictureInPicture];
    // 回调画中画结束
    [_delegate onStopPictureInPicture];
}

- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController
    restoreUserInterfaceForPictureInPictureStopWithCompletionHandler:(void (^)(BOOL restored))completionHandler {
    completionHandler(true);
    // 恢复全屏界面
}

4、更新缓冲区图层

在ijk解码出来后，封装成CVPixelBuffer，然后封装CMSampleBufferRef，存入displayLayer。示例代码如下：

-(void) enqueueBuffer:(CVPixelBufferRef)pixelBuffer
{
    if (!_displayLayer || !pixelBuffer)
        return;
    @autoreleasepool {
        CMSampleBufferRef sampleBuffer = nil;
        CMVideoFormatDescriptionRef format = nil;
        // 设置dts、pts
        CMSampleTimingInfo timeInfo = {.presentationTimeStamp = kCMTimeInvalid, .decodeTimeStamp = kCMTimeInvalid};
        OSStatus status = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, &format);
        status = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, true, NULL, NULL,
                                                    format, &timeInfo, &sampleBuffer);
        if (format != nil) {
            CFRelease(format);
        }
        if (sampleBuffer == nil || status != noErr) {
            NSLog(@"sampleBuffer error!");
            return;
        }
        CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, YES);
        CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);
        // 设置立即渲染
        CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);
        
        [_displayLayer enqueueSampleBuffer:sampleBuffer];
        CFRelease(sampleBuffer);
        if (_displayLayer.status == AVQueuedSampleBufferRenderingStatusFailed) {
            NSLog(@"displayLayer error=%@", [_displayLayer error]);
        }
    }
}

5、更新播放状态

实现AVPictureInPictureSampleBufferPlaybackDelegate代理方法，更新播放状态、seek进度。示例代码如下：

- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController setPlaying:(BOOL)playing {
    _isPlaybackPaused = !playing;
    // 回调播放/暂停
    [_delegate setPlaying:playing];
    if (@available(iOS 15.0, *)) {
        [pictureInPictureController invalidatePlaybackState];
    }
    if (playing == NO) {
        _playClickTime = [[NSDate date] timeIntervalSince1970] * 1000;
    }
}

- (CMTimeRange)pictureInPictureControllerTimeRangeForPlayback:(AVPictureInPictureController *)pictureInPictureController {
    // 计算time range：0～duration
    NSTimeInterval durationTime = [_delegate getDurationTime];
    return CMTimeRangeMake(kCMTimeZero, CMTimeMake(durationTime * 1000, 1000));
}

- (BOOL)pictureInPictureControllerIsPlaybackPaused:(AVPictureInPictureController *)pictureInPictureController {
    return _isPlaybackPaused;
}

- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController
         didTransitionToRenderSize:(CMVideoDimensions)newRenderSize {

}

- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController skipByInterval:(CMTime)skipInterval
                 completionHandler:(void (^)(void))completionHandler {
    // 回调seek
    [_delegate seekTo:CMTimeGetSeconds(skipInterval)];
    if (@available(iOS 15.0, *)) {
        // seek开始会暂停播放，seek成功后恢复播放状态
        if (([[NSDate date] timeIntervalSince1970] * 1000 - _playClickTime) < 200) {
            [_delegate setPlaying:YES];
        }
        // 更新displaylayer的controlTimebase时钟，从而更新进度
        CMTimebaseSetTime(_displayLayer.controlTimebase, CMTimeMake([_delegate getCurrentTime] * 1000, 1000));
        // 更新播放状态
        [_pipController invalidatePlaybackState];
    }
}

ijkplayer提供play、seek、getCurrentPosition、getDuration、isPlaying等接口，给自定义的画中画Controller调用：

- (BOOL)isPlaybackPaused
{
    return [self isPlaying] == NO;
}

- (void)setPlaying:(BOOL)playing
{
    if (playing) {
        [self play];
    } else {
        [self pause];
    }
}

- (void)seekTo:(NSTimeInterval)relativePosition
{
    NSTimeInterval position = [self currentPlaybackTime] + relativePosition;
    [self setCurrentPlaybackTime:position];
}

- (NSTimeInterval)getCurrentTime
{
    return [self currentPlaybackTime];
}

- (NSTimeInterval)getDurationTime
{
    return [self duration];
}