SurfaceTexture OnFrameAvailableListener 调用流程分析

背景:

最近项目中遇到一个问题, 需要搞清楚OnFrameAvailableListener 回调流程, 本文借此机会做个记录, 巩固印象, 有相关困惑的同学也可以参考下.

本文基于Android 14 framework 源码进行分析

SurfaceTexture.java

OnFrameAvailableListener 设置过程

 public void setOnFrameAvailableListener(@Nullable final OnFrameAvailableListener listener,
            @Nullable Handler handler) {
        if (listener != null) {
            // Although we claim the thread is arbitrary, earlier implementation would
            // prefer to send the callback on the creating looper or the main looper
            // so we preserve this behavior here.
            Looper looper = handler != null ? handler.getLooper() :
                    mCreatorLooper != null ? mCreatorLooper : Looper.getMainLooper();
            mOnFrameAvailableHandler = new Handler(looper, null, true /*async*/) {
                @Override
                public void handleMessage(Message msg) {
                    listener.onFrameAvailable(SurfaceTexture.this);
                }
            };
        } else {
            mOnFrameAvailableHandler = null;
        }
    }

一直很好奇,硬件解码后回调是怎么调用的, 首先java在设置回调的时候,回看调用方是不是有handler, 如果没有handler 会发送到主线程执行, 如果有handler,就会把回调发送这个handler执行,接下来来看下这个mOnFrameAvailableHandler在什么地方调用

mOnFrameAvailableHandler

private static void postEventFromNative(WeakReference<SurfaceTexture> weakSelf) {
    SurfaceTexture st = weakSelf.get();
    if (st != null) {
        Handler handler = st.mOnFrameAvailableHandler;
        if (handler != null) {
            handler.sendEmptyMessage(0);
        }
    }
}

native 会调用这个方法, 然后这个方法在handler里边发送消息,通知回调,接下来看下native是怎么调用的

postEventFromNative

static void SurfaceTexture_classInit(JNIEnv* env, jclass clazz)
{
   
    fields.postEvent = env->GetStaticMethodID(clazz, "postEventFromNative",
            "(Ljava/lang/ref/WeakReference;)V");
    if (fields.postEvent == NULL) {
        ALOGE("can't find android/graphics/SurfaceTexture.postEventFromNative");
    }
}
class JNISurfaceTextureContextCommon {
public:
    JNISurfaceTextureContextCommon(JNIEnv* env, jobject weakThiz, jclass clazz)
          : mWeakThiz(env->NewGlobalRef(weakThiz)), mClazz((jclass)env->NewGlobalRef(clazz)) {}
    ...
    void onFrameAvailable(const BufferItem& item) {
        JNIEnv* env = getJNIEnv();
        if (env != NULL) {
            env->CallStaticVoidMethod(mClazz, fields.postEvent, mWeakThiz);
        } else {
            ALOGW("onFrameAvailable event will not posted");
        }
    }

protected:
    ....

    jobject mWeakThiz;
    jclass mClazz;
};
class JNISurfaceTextureContextFrameAvailableListener
      : public JNISurfaceTextureContextCommon,
        public SurfaceTexture::FrameAvailableListener {
public:
    JNISurfaceTextureContextFrameAvailableListener(JNIEnv* env, jobject weakThiz, jclass clazz)
          : JNISurfaceTextureContextCommon(env, weakThiz, clazz) {}
    void onFrameAvailable(const BufferItem& item) override {
        JNISurfaceTextureContextCommon::onFrameAvailable(item);
    }
};

JNISurfaceTextureContextFrameAvailableListener 集成了JNISurfaceTextureContextCommon,SurfaceTexture::FrameAvailableListener, 并实现了方法onFrameAvailable,再通过jni去调用java的postEventFromNative, 接下来看 JNISurfaceTextureContextFrameAvailableListener 设置到那里的

JNISurfaceTextureContextFrameAvailableListener

static void SurfaceTexture_init(JNIEnv* env, jobject thiz, jboolean isDetached,
        jint texName, jboolean singleBufferMode, jobject weakThiz)
{
    sp<IGraphicBufferProducer> producer;
    sp<IGraphicBufferConsumer> consumer;
    BufferQueue::createBufferQueue(&producer, &consumer);

    
    sp<SurfaceTexture> surfaceTexture;
   
      surfaceTexture = new SurfaceTexture(consumer, texName,
              GL_TEXTURE_EXTERNAL_OES, true, !singleBufferMode);


   
    surfaceTexture->setName(String8::format("SurfaceTexture-%d-%d-%d",
            (isDetached ? 0 : texName),
            getpid(),
            createProcessUniqueId()));

    SurfaceTexture_setSurfaceTexture(env, thiz, surfaceTexture);
    SurfaceTexture_setProducer(env, thiz, producer);

    jclass clazz = env->GetObjectClass(thiz);

    sp<JNISurfaceTextureContextFrameAvailableListener> ctx(
            new JNISurfaceTextureContextFrameAvailableListener(env, weakThiz, clazz));
    surfaceTexture->setFrameAvailableListener(ctx);
    SurfaceTexture_setFrameAvailableListener(env, thiz, ctx);
 
}

看的出来SurfaceTexture在初始化时候, 会创建生产者(producer),与消费者(consumer), 接着会把ctx设置到surfaceTexture->setFrameAvailableListener, 接下来看下内部代码

surfaceTexture::setFrameAvailableListener

SurfaceTexture 继承了ConsumerBase, SurfaceTexture并没有实现setFrameAvailableListener, 在其父类实现的,接下来看下父类的代码


class ConsumerListener : public virtual RefBase {
public:
    ConsumerListener() {}
    virtual ~ConsumerListener();
    ...
    virtual void onFrameAvailable(const BufferItem& item) = 0;
    ...

}
ConsumerBase::ConsumerBase(const sp<IGraphicBufferConsumer>& bufferQueue, bool controlledByApp) :
        mAbandoned(false),
        mConsumer(bufferQueue),
        mPrevFinalReleaseFence(Fence::NO_FENCE) {
      ...
       wp<ConsumerListener> listener = static_cast<ConsumerListener*>(this);
      status_t err = mConsumer->consumerConnect(proxy, controlledByApp);
      ...
}

void ConsumerBase::setFrameAvailableListener(
        const wp<FrameAvailableListener>& listener) {
    CB_LOGV("setFrameAvailableListener");
    Mutex::Autolock lock(mFrameAvailableMutex);
    mFrameAvailableListener = listener;
}

void ConsumerBase::onFrameAvailable(const BufferItem& item) {

    sp<FrameAvailableListener> listener;
    { // scope for the lock
        Mutex::Autolock lock(mFrameAvailableMutex);
        listener = mFrameAvailableListener.promote();
    }

    if (listener != nullptr) {
        listener->onFrameAvailable(item);
    }
}

ConsumerBase 继承了ConsumerListener, 并重写了onFrameAvailable, 在ConsumerBase 构造调用了consumerConnect, 这个方法很关键,接下来看下这个方法在干什么

BufferQueueConsumer::consumerConnect

class BufferQueueConsumer : public BnGraphicBufferConsumer {
   virtual status_t consumerConnect(const sp<IConsumerListener>& consumer,
            bool controlledByApp) {
        return connect(consumer, controlledByApp);
    }
}
status_t BufferQueueConsumer::connect(
        const sp<IConsumerListener>& consumerListener, bool controlledByApp) {

    std::lock_guard<std::mutex> lock(mCore->mMutex);

    mCore->mConsumerListener = consumerListener;
    mCore->mConsumerControlledByApp = controlledByApp;

    return NO_ERROR;
}

这里就很清晰了, consumerConnect 会把上边的回调ConsumerBase::onFrameAvailable 回调设置到mConsumer(BufferQueueConsumer) 的 mCore->mConsumerListener中去,

我们知道BufferQueueProducer与BufferQueueConsumer共享一个BufferQueueCore 如下图所示:

也就说到这里生成者Producer通过Core就可以获取到Consumer的回调了

接下来看下回调是怎么调用的

回调调用

我们已经知道了SurfaceTexture 创建成功后,还需要创建Surface作为其生产者, 接下来从生产者的角度看下回调调用流程是怎么样的.

Surface作为生产者在dequeuebuffer获取一帧数据后, 在填充后, 会调用queuebuffer,我们就从这个queuebuffer 去跟下源码

int Surface::hook_queueBuffer(ANativeWindow* window,
        ANativeWindowBuffer* buffer, int fenceFd) {
    Surface* c = getSelf(window);
    {
        std::shared_lock<std::shared_mutex> lock(c->mInterceptorMutex);
        if (c->mQueueInterceptor != nullptr) {
            auto interceptor = c->mQueueInterceptor;
            auto data = c->mQueueInterceptorData;
            return interceptor(window, Surface::queueBufferInternal, data, buffer, fenceFd);
        }
    }
    return c->queueBuffer(buffer, fenceFd);
}


int Surface::queueBuffer(android_native_buffer_t* buffer, int fenceFd) {
    ATRACE_CALL();
    ALOGV("Surface::queueBuffer");
    Mutex::Autolock lock(mMutex);

    int i = getSlotFromBufferLocked(buffer);
   
    IGraphicBufferProducer::QueueBufferOutput output;
    IGraphicBufferProducer::QueueBufferInput input;
    getQueueBufferInputLocked(buffer, fenceFd, mTimestamp, &input);
    applyGrallocMetadataLocked(buffer, input);
    sp<Fence> fence = input.fence;

    nsecs_t now = systemTime();

    status_t err = mGraphicBufferProducer->queueBuffer(i, input, &output);
   

    onBufferQueuedLocked(i, fence, output);
    return err;
}

status_t BufferQueueProducer::queueBuffer(int slot,
        const QueueBufferInput &input, QueueBufferOutput *output) {
    ATRACE_CALL();
    ATRACE_BUFFER_INDEX(slot);

      //...
      frameAvailableListener = mCore->mConsumerListener;
      //..
      if (frameAvailableListener != nullptr) {
            frameAvailableListener->onFrameAvailable(item);
      } 
      //..
    } // Autolock scope

这个BufferQueueProducer::queueBuffer 代码很多, 这里删除了无关代码, 看到在queueBuffer 入队的时候会获取mCore->mConsumerListener, 这就是上边ConsumerBase::onFrameAvailable回调, 调用这个最终就会到消费者回调中去,从而一层一层通知,最终到SurfaceTexture.java 往handler发送消息, 最终回调出去