渲染视频画面
======

#### 概述

渲染视频画面是将原始视频数据（如 YUV、RGB 等格式）转换为可在屏幕上显示的图像，并最终输出到显示设备（如手机屏幕）的过程。此过程是视频解码播放、视频直播、相机预览等场景中的关键环节，决定了画面视觉效果。

本文以视频解码播放为场景案例，介绍了视频在解码后，如何将解码的视频数据渲染送显到设备屏幕上。目前，系统中提供了多种方式渲染视频画面，包括使用XComponent渲染视频画面、使用OpenGL渲染视频画面、使用Vulkan渲染视频画面。这三个方案的优点、缺点和适用场景如下表所示。

|    | 
优点

 | 

缺点

 | 

使用场景

 |
| :-- | :-- | :-- | :-- |
| 

使用XComponent渲染视频画面

 | 

*   开发门槛不高，可以通过XComponent的NativeWindow直接对接解码器。
*   解码后的YUV视频数据，无需额外进行处理，可以直接提交给XComponent进行渲染显示。

 | 

*   定制化能力弱，无法满足复杂的开发场景。
*   性能优化的空间不多，在较为复杂或高规格的场景中，无法干预渲染流程，导致性能不高。

 | 

该方案适用于简单的视频播放，不需要对视频画面做复杂的处理。

 |
| 

使用OpenGL渲染视频画面

 | 

*   可以满足复杂的场景，根据开发者的需求定制化开发。
*   OpenGL是跨编程语言和跨平台的应用编程接口，便于应用的开发与迁移。
*   开发门槛适中，生态资料丰富。

 | 

*   需要额外处理视频数据的格式，视频解码后的数据通常需要转化成RGBA的格式。

 | 

该方案可以满足复杂的开发场景，开发者可以根据实际需求实现，如视频直播、视频弹幕等场景。

 |
| 

使用Vulkan渲染视频画面

 | 

*   可细粒度控制渲染全流程，优化渲染链路以获得更高性能。
*   可适配复杂的场景。
*   Vulkan是跨编程语言和跨平台的应用编程接口，便于应用的开发与迁移。

 | 

*   需要额外处理视频数据的格式，视频解码后的数据通常需要转化成RGBA的格式。
*   开发门槛较高，需要手动管理内存、命令缓冲等，且需要开发的代码量大，出现问题难定位。

 | 

同上。

 |

#### 使用XComponent渲染视频画面

#### \[h2\]场景描述

系统提供了AVPlayer、Video组件播放视频文件，其实现方式简单，但支持的文件格式有限。在播放其他格式（如rmvb格式）或有数字版权保护的视频时，AVPlayer、Video组件不能满足开发者的诉求。此时，开发者可以通过AVCodec将视频文件进行解码，然后将解码后的视频数据通过XComponent组件直接进行渲染送显即可。使用XComponent渲染视频画面是简单、常用的方式，开发者可以将解码的YUV视频数据传入给XComponent，无需关注YUV格式转化等问题。

#### \[h2\]实现原理

在使用XComponent渲染视频画面的方案中，需要提前获取XComponent对应的NativeWindow对象，并在创建视频解码器时，将NativeWindow设置给视频解码器。其具体开发流程如下所示。

1.  在XComponent创建时，通过回调函数OnSurfaceCreatedCB获取对应的NativeWindow对象。
2.  初始化视频解码的环境，包括初始化解封装器、初始化解码器。
3.  启动解码器、解码输入子线程、解码输出子线程。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/f2/v3/89jcVHeRT1Kj4GV_LyP8EQ/zh-cn_image_0000002484614821.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=04D9FB4CFB4057F0A72DA872DAF65E3972D83DB41497CB556D59735514E0D821 "点击放大")
    
4.  通过OnNeedInputBuffer获取可用的AVBuffer后，在解码输入子线程中，将解封装器读取视频数据填充到AVBuffer中，并提交给解码器进行解码。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/69/v3/dJTactmOR76K4GsFxvyiLA/zh-cn_image_0000002451534962.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=B3D74523AFC0AA012B7FFEC22BB586E325E8F93E3505FFCC533C25C29A3598FF "点击放大")
    
5.  通过OnNeedOutputBuffer获取解码的视频数据后，在解码输出子线程中，将解码后的视频数据提交给输出Surface（即XComponent的NativeWindow）。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/f5/v3/lkShhNUjRd6arE7qR_6_Qw/zh-cn_image_0000002451375366.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=4C401D85EA3D99F8B24939BE43AF3B3BF4195ED28CFEBADD21C35016A414DD36 "点击放大")
    

#### \[h2\]开发步骤

1.  在XComponent创建时，通过回调函数OnSurfaceCreatedCB获取对应的NativeWindow对象。
    
    void OnSurfaceCreatedCB(OH\_NativeXComponent \*component, void \*window)
    {
        OH\_LOG\_Print(LOG\_APP, LOG\_INFO, LOG\_PRINT\_DOMAIN, "Callback", "OnSurfaceCreatedCB");
        if ((component == nullptr) || (window == nullptr)) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "Callback",
                         "onSurfaceCreatedCB: component or window is null");
            return;
        }
        char idStr\[OH\_XCOMPONENT\_ID\_LEN\_MAX + 1\] = {'\\0'};
        uint64\_t idSize = OH\_XCOMPONENT\_ID\_LEN\_MAX + 1;
        if (OH\_NativeXComponent\_GetXComponentId(component, idStr, &idSize) != OH\_NATIVEXCOMPONENT\_RESULT\_SUCCESS) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "Callback",
                         "OnSurfaceCreatedCB:Unable to get XComponent id");
            return;
        }
        std::string id(idStr);
        auto render = PluginRender::GetInstance(id);
        uint64\_t width;
        uint64\_t height;
        int32\_t xSize = OH\_NativeXComponent\_GetXComponentSize(component, window, &width, &height);
        if ((xSize != OH\_NATIVEXCOMPONENT\_RESULT\_SUCCESS) || (render == nullptr)) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "Callback",
                         "OnSurfaceCreatedCB: Unable to qet XComponent size");
            return;
        }
        render->nativeWindow = reinterpret\_cast<OHNativeWindow \*>(window);
        (void)OH\_NativeWindow\_NativeWindowHandleOpt(render->nativeWindow, SET\_BUFFER\_GEOMETRY, static\_cast<int>(width),
                                                        static\_cast<int>(height));
        if (id == "OpenGL") {
            render->openGLRenderThread\_->UpdateNativeWindow(render->nativeWindow, width, height);
        } else if(id == "Vulkan") {
            render->vulkanRenderThread\_->UpdateNativeWindow(render->nativeWindow, width, height);
        }
    }
    
2.  初始化视频解码的环境。
    
    int32\_t Player::Init(SampleInfo &sampleInfo) {
        std::unique\_lock<std::mutex> lock(mutex\_);
        if(isStarted\_ || (demuxer\_ != nullptr && videoDecoder\_ != nullptr)) {
            OH\_LOG\_ERROR(LOG\_APP, "Already started.");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        sampleInfo\_ = sampleInfo;
    
        videoDecoder\_ = std::make\_unique<VideoDecoder>();
        demuxer\_ = std::make\_unique<Demuxer>();
        isReleased\_ = false;
        int32\_t ret = demuxer\_->Create(sampleInfo\_);
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            OH\_LOG\_ERROR(LOG\_APP, "Create demuxer failed");
            doneCond\_.notify\_all();
            lock.unlock();
            StartRelease();
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        ret = CreateVideoDecoder();
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            OH\_LOG\_ERROR(LOG\_APP, "Create video decoder failed");
            doneCond\_.notify\_all();
            lock.unlock();
            StartRelease();
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        OH\_LOG\_INFO(LOG\_APP, "Succeed");
        return AVCODEC\_SAMPLE\_ERR\_OK;
    }
    
    创建解封装器。
    
    int32\_t Demuxer::Create(SampleInfo &info) {
        source\_ = OH\_AVSource\_CreateWithFD(info.inputFd, info.inputFileOffset, info.inputFileSize);
        if (source\_ == nullptr) {
            OH\_LOG\_ERROR(LOG\_APP,
                         "Create demuxer source failed, fd: %{public}d, offset: %{public}ld, file size: %{public}ld",
                         info.inputFd, info.inputFileOffset, info.inputFileSize);
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
        
        demuxer\_ = OH\_AVDemuxer\_CreateWithSource(source\_);
        if (demuxer\_ == nullptr) {
            OH\_LOG\_ERROR(LOG\_APP, "Create demuxer failed");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        auto sourceFormat = std::shared\_ptr<OH\_AVFormat>(OH\_AVSource\_GetSourceFormat(source\_), OH\_AVFormat\_Destroy);
        if (sourceFormat == nullptr) {
            OH\_LOG\_ERROR(LOG\_APP, "Get source format failed");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        int32\_t ret = GetTrackInfo(sourceFormat, info);
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            OH\_LOG\_ERROR(LOG\_APP, "Get video track info, ret: %{public}d", ret);
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
        
        return AVCODEC\_SAMPLE\_ERR\_OK;
    }
    
    创建视频解码器。
    
    int32\_t Player::CreateVideoDecoder() {
        OH\_LOG\_INFO(LOG\_APP, "video mime:%{public}s", sampleInfo\_.videoCodecMime.c\_str());
        int32\_t ret = videoDecoder\_->Create(sampleInfo\_.videoCodecMime);
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            OH\_LOG\_INFO(LOG\_APP, "Create video decoder failed, mime:%{public}s", sampleInfo\_.videoCodecMime.c\_str());
        } else {
            videoDecContext\_ = new CodecUserData;
            ret = videoDecoder\_->Config(sampleInfo\_, videoDecContext\_);
            if(!(ret == AVCODEC\_SAMPLE\_ERR\_OK)) {
                OH\_LOG\_ERROR(LOG\_APP, "Video Decoder config failed");
                return ret;
            }
        }
        return AVCODEC\_SAMPLE\_ERR\_OK;
    }
    
    设置视频解码器的Surface。
    
    int32\_t VideoDecoder::Config(const SampleInfo &sampleInfo, CodecUserData \*codecUserData) {
        if (decoder\_ == nullptr) {
            OH\_LOG\_ERROR(LOG\_APP, "Decoder is null");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        if (codecUserData == nullptr) {
            OH\_LOG\_ERROR(LOG\_APP, "Invalid param: codecUserData");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
        
        int32\_t ret = Configure(sampleInfo);
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            OH\_LOG\_ERROR(LOG\_APP, "Configure failed");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        if (sampleInfo.window != nullptr) {
            int ret = OH\_VideoDecoder\_SetSurface(decoder\_, sampleInfo.window);
            if (ret != AV\_ERR\_OK) {
                OH\_LOG\_ERROR(LOG\_APP, "Set surface failed, ret: %{public}d", ret);
                return AVCODEC\_SAMPLE\_ERR\_ERROR;
            }
        }
        
        ret = SetCallback(codecUserData);
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            OH\_LOG\_ERROR(LOG\_APP, "Set callback failed, ret: %{public}d", ret);
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
        
        ret = OH\_VideoDecoder\_Prepare(decoder\_);
        if (ret != AV\_ERR\_OK) {
            OH\_LOG\_ERROR(LOG\_APP, "Prepare failed, ret: %{public}d", ret);
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        return AVCODEC\_SAMPLE\_ERR\_OK;
    }
    
3.  启动解码器、解码输入子线程、解码输出子线程。
    
    int32\_t Player::Start() {
        std::unique\_lock<std::mutex> lock(mutex\_);
        int32\_t ret;
        if (isStarted\_ || demuxer\_ == nullptr) {
            OH\_LOG\_ERROR(LOG\_APP, "Already started.");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
        
        if (videoDecContext\_) {
            ret = videoDecoder\_->Start();
            if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
                OH\_LOG\_ERROR(LOG\_APP, "Video Decoder start failed");
                lock.unlock();
                StartRelease();
                return AVCODEC\_SAMPLE\_ERR\_ERROR;
            }
            isStarted\_ = true;
            videoDecInputThread\_ = std::make\_unique<std::thread>(&Player::VideoDecInputThread, this);
            videoDecOutputThread\_ = std::make\_unique<std::thread>(&Player::VideoDecOutputThread, this);
    
            if (videoDecInputThread\_ == nullptr || videoDecOutputThread\_ == nullptr) {
                OH\_LOG\_ERROR(LOG\_APP, "Create thread failed");
                lock.unlock();
                StartRelease();
                return AVCODEC\_SAMPLE\_ERR\_ERROR;
            }
        }
    
        OH\_LOG\_INFO(LOG\_APP, "Succeed");
        doneCond\_.notify\_all();
        return AVCODEC\_SAMPLE\_ERR\_OK;
    }
    
4.  通过OnNeedInputBuffer获取可用的AVBuffer，将AVBuffer放到队列中，便于后续处理。
    
    void SampleCallback::OnNeedInputBuffer(OH\_AVCodec \*codec, uint32\_t index, OH\_AVBuffer \*buffer, void \*userData) {
        if (userData == nullptr) {
            return;
        }
        (void)codec;
        CodecUserData \*codecUserData = static\_cast<CodecUserData \*>(userData);
        std::unique\_lock<std::mutex> lock(codecUserData->inputMutex);
        codecUserData->inputBufferInfoQueue.emplace(index, buffer);
        codecUserData->inputCond.notify\_all();
    }
    
    在解码输入子线程中，将解封装器读取视频数据填充到AVBuffer中，并提交给解码器进行解码。
    
    void Player::VideoDecInputThread() {
        while (true) {
            if (!isStarted\_) {
                OH\_LOG\_ERROR(LOG\_APP, "Decoder input thread out");
                break;;
            }
            
            std::unique\_lock<std::mutex> lock(videoDecContext\_->inputMutex);
            bool condRet = videoDecContext\_->inputCond.wait\_for(
                lock, 5s, \[this\]() { return !isStarted\_ || !videoDecContext\_->inputBufferInfoQueue.empty(); });
            if (!isStarted\_) {
                OH\_LOG\_ERROR(LOG\_APP, "Work done, thread out");
                break;
            }
            if (videoDecContext\_->inputBufferInfoQueue.empty()) {
                OH\_LOG\_ERROR(LOG\_APP, "Buffer queue is empty, continue, cond ret: %{public}d", condRet);
                continue;
            }
    
            CodecBufferInfo bufferInfo = videoDecContext\_->inputBufferInfoQueue.front();
            videoDecContext\_->inputBufferInfoQueue.pop();
            videoDecContext\_->inputFrameCount++;
            lock.unlock();
    
            demuxer\_->ReadSample(demuxer\_->GetVideoTrackId(), reinterpret\_cast<OH\_AVBuffer \*>(bufferInfo.buffer),
                                 bufferInfo.attr);
    
            int32\_t ret = videoDecoder\_->PushInputBuffer(bufferInfo);
            if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
                OH\_LOG\_ERROR(LOG\_APP, "Push data failed, thread out");
                break;
            }
    
            if (bufferInfo.attr.flags & AVCODEC\_BUFFER\_FLAGS\_EOS) {
                OH\_LOG\_ERROR(LOG\_APP, "Catch EOS, thread out");
                break;
            }
        }
    }
    
5.  在解码输出子线程中，将解码后的视频数据提交给输出Surface。通过OnNeedOutputBuffer获取解码的视频数据。
    
    void SampleCallback::OnNewOutputBuffer(OH\_AVCodec \*codec, uint32\_t index, OH\_AVBuffer \*buffer, void \*userData) {
        if (userData == nullptr) {
            return;
        }
        (void)codec;
        CodecUserData \*codecUserData = static\_cast<CodecUserData \*>(userData);
        std::unique\_lock<std::mutex> lock(codecUserData->outputMutex);
        codecUserData->outputBufferInfoQueue.emplace(index, buffer);
        codecUserData->outputCond.notify\_all();
    }
    
    在解码输出子线程中，将解码后的视频数据提交给输出Surface。Surface在已在解码器初始化配置，设置的是XComponent的NativeWindow对象。
    
    void Player::VideoDecOutputThread() {
        sampleInfo\_.frameInterval = MICROSECOND / sampleInfo\_.frameRate;
        while (true) {
            thread\_local auto lastPushTime = std::chrono::system\_clock::now();
            if (!isStarted\_) {
                OH\_LOG\_ERROR(LOG\_APP, "Decoder output thread out");
                break;
            }
            std::unique\_lock<std::mutex> lock(videoDecContext\_->outputMutex);
            bool condRet = videoDecContext\_->outputCond.wait\_for(
                lock, 5s, \[this\]() { return !isStarted\_ || !videoDecContext\_->outputBufferInfoQueue.empty(); });
            if (!isStarted\_) {
                OH\_LOG\_ERROR(LOG\_APP, "Decoder output thread out");
                break;
            }
            if (videoDecContext\_->outputBufferInfoQueue.empty()) {
                OH\_LOG\_ERROR(LOG\_APP, "Buffer queue is empty, continue, cond ret: %{public}d", condRet);
                continue;
            }
    
            CodecBufferInfo bufferInfo = videoDecContext\_->outputBufferInfoQueue.front();
            videoDecContext\_->outputBufferInfoQueue.pop();
            if (bufferInfo.attr.flags & AVCODEC\_BUFFER\_FLAGS\_EOS) {
                OH\_LOG\_ERROR(LOG\_APP, "Catch EOS, thread out");
                break;
            }
            videoDecContext\_->outputFrameCount++;
            OH\_LOG\_INFO(LOG\_APP,"Out buffer count: %{public}u, size: %{public}d, flag: %{public}u, pts: %{public}ld",
                                videoDecContext\_->outputFrameCount, bufferInfo.attr.size, bufferInfo.attr.flags,
                                bufferInfo.attr.pts);
            lock.unlock();
    
            int32\_t ret = videoDecoder\_->FreeOutputBuffer(bufferInfo.bufferIndex, true);
            if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
                OH\_LOG\_ERROR(LOG\_APP, "Decoder output thread out");
                break;
            }
    
            std::this\_thread::sleep\_until(lastPushTime + std::chrono::microseconds(sampleInfo\_.frameInterval));
            lastPushTime = std::chrono::system\_clock::now();
        }
        StartRelease();
    }
    
    通过OH\_VideoDecoder\_RenderOutputBuffer通知解码器将视频数据提交给输出Surface。
    
    int32\_t VideoDecoder::FreeOutputBuffer(uint32\_t bufferIndex, bool render) {
        if (decoder\_ == nullptr) {
            OH\_LOG\_ERROR(LOG\_APP, "Decoder is null");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
    
        int32\_t ret = AVCODEC\_SAMPLE\_ERR\_OK;
        if (render) {
            ret = OH\_VideoDecoder\_RenderOutputBuffer(decoder\_, bufferIndex);
        } else {
            ret = OH\_VideoDecoder\_FreeOutputBuffer(decoder\_, bufferIndex);
        }
        if (ret != AV\_ERR\_OK) {
            OH\_LOG\_ERROR(LOG\_APP, "Free output data failed");
            return AVCODEC\_SAMPLE\_ERR\_ERROR;
        }
        return AVCODEC\_SAMPLE\_ERR\_OK;
    }
    

#### 使用OpenGL渲染视频画面

#### \[h2\]场景描述

OpenGL(Open Graphics Library)是一种跨语言、跨平台的应用程序编程接口（API），用于渲染2D和3D矢量图形。使用XComponent渲染视频画面的实现方案简单，支持的视频格式多，但对视频画面处理不够灵活，例如视频放大、旋转等操作。通过OpenGL，开发者可以根据需求定制实现各种图像处理功能，如视频旋转、视频弹幕等。

#### \[h2\]实现原理

在使用OpenGL渲染视频画面中，需要使用NativeImage对接解码器和OpenGL。开发者需要通过OH\_NativeImage\_AcquireNativeWindow()获取NativeImage的NativeWindow对象，并将NativeImage的NativeWindow对象设置给解码器，以消费解码器生成的视频图像数据。然后，通过OH\_NativeImage\_UpdateSurfaceImage()将NativeImage获取到的视频图像缓存更新到Opengl相关的纹理中。最后，通过eglSwapBuffers将渲染的缓存提交给XComponent的NativeWindow对象。

其具体实现如下所示。

1.  启动渲染子线程，初始化OpenGL环境，包括获取显示设备、初始化EGL环境、创建OpenGL上下文等。
2.  创建NativeImage对象，并根据NativeImage获取NativeWindow对象。
3.  获取XComponent的NativeWindow对象，并根据XComponent的NativeWindow对象创建EGLSurface。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/0a/v3/QuOhwxraS7SVZdKf5vLvQw/zh-cn_image_0000002484494809.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=CD383E4F54457ABF7B0A37048CC26EB12C97ACBF72B9F75E3240BAC0E10DFE64 "点击放大")
    
4.  初始化视频解码的环境，包括初始化解封装器、初始化配置解码器。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/0d/v3/F88pFJQRRUC2V33iRlbxxA/note_3.0-zh-cn.png?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=36BE2BC2AEB471DF7F67CD95D1C8A8D7FBC2258186B6C360AEB167E82275D7C6)
    
    在初始化配置解码器时，与直接使用XComponent渲染的方案不同，解码器设置Surface的入参是NativeImage的NativeWindow对象，而不是XComponent的NativeWindow对象。
    
5.  启动解码器、解码输入子线程、解码输出子线程。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/9f/v3/vLW4FHTZSAeXoCDSG3aG9g/zh-cn_image_0000002484614829.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=F150254CE9B5ABFE44543DC0997504DE171F067A6B5E84012AFDC60F3200CF3E "点击放大")
    
6.  通过OnNeedInputBuffer获取可用的AVBuffer后，在解码输入子线程中，将解封装器读取视频数据填充到AVBuffer中，并提交给解码器进行解码。
7.  通过OnNeedOutputBuffer获取解码的视频数据后，在解码输出子线程中，将解码后的视频数据提交给输出Surface（当前Surface为NativeImage的NativeWindow对象）。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/ea/v3/6uGu9ZsOSIuFVbN7kfw2CQ/zh-cn_image_0000002451534970.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=2C0255F6600BE6CD263A8F8A64AA2A3C1E08F80E1169A8D69B9F8DA076266660 "点击放大")
    
8.  通过NativeImage，将视频图像缓存更新至OpenGL的纹理上。
9.  通过eglSwapBuffers交换前后缓冲区的内容，并将渲染结果显示在屏幕。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/31/v3/wDFpsnFATVeziRp0_Y0-Jw/zh-cn_image_0000002451375374.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=86DBBD0EA027B21304BF427E637BE666C7F385CF7E64591DBFCD25E488AE2339 "点击放大")
    

#### \[h2\]开发步骤

1.  启动渲染子线程，初始化OpenGL环境。
    
    void OpenGLRenderThread::ThreadMainLoop()
    {
        threadId\_ = std::this\_thread::get\_id();
        if (!InitRenderContext()) {
            return;
        }
        if (!InitNativeVsync()) {
            return;
        }
        if (!CreateNativeImage()) {
            return;
        }
        while (running\_) {
            {
                std::unique\_lock<std::mutex> lock(wakeUpMutex\_);
                wakeUpCond\_.wait(lock, \[this\]() { return wakeUp\_ || vSyncCnt\_ > 0 || availableFrameCnt\_ > 0; });
                wakeUp\_ = false;
                vSyncCnt\_--;
                (void)OH\_NativeVSync\_RequestFrame(nativeVsync\_, &OpenGLRenderThread::OnVsync, this);
            }
            
            std::vector<RenderTask> tasks;
            {
                std::unique\_lock<std::mutex> lock(taskMutex\_);
                tasks.swap(tasks\_);
            }
            for (const auto &task : tasks) {
                task(\*renderContext\_);
            }
    
            if (availableFrameCnt\_ <= 0) {
                continue;
            }
            DrawImage();
            availableFrameCnt\_--;
        }
    }
    
    初始化OpenGL环境，包括获取Display、EGL初始化、EGL上下文的创建等。
    
    bool OpenGLRender::Init()
    {
        if (IsEglContextReady()) {
            return true;
        }
    
        OH\_LOG\_Print(LOG\_APP, LOG\_DEBUG, LOG\_PRINT\_DOMAIN, "EglRenderContext", "EglRenderContext::Init begin.");
        eglDisplay\_ = GetPlatformEglDisplay(EGL\_PLATFORM\_OHOS\_KHR, EGL\_DEFAULT\_DISPLAY, NULL);
        if (eglDisplay\_ == EGL\_NO\_DISPLAY) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "EglRenderContext",
                         "EglRenderContext::Init: failed to create eglDisplay, error: %{public}s.", GetEglErrorString());
            return false;
        }
    
        EGLint major;
        EGLint minor;
        if (eglInitialize(eglDisplay\_, &major, &minor) == EGL\_FALSE) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "EglRenderContext",
                         "EglRenderContext::Init: Failed to initialize EGLDisplay, error: %{public}s.",
                         GetEglErrorString());
        }
        SetupEglExtensions();
    
        if (eglBindAPI(EGL\_OPENGL\_ES\_API) == EGL\_FALSE) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "EglRenderContext",
                         "EglRenderContext::Init: Failed to bind OpenGL ES API, error: %{public}s.", GetEglErrorString());
            return false;
        }
        EGLint count;
        EGLint configAttribs\[\] = { EGL\_SURFACE\_TYPE,
                                   EGL\_WINDOW\_BIT,
                                   EGL\_RED\_SIZE,
                                   8,
                                   EGL\_GREEN\_SIZE,
                                   8,
                                   EGL\_BLUE\_SIZE,
                                   8,
                                   EGL\_ALPHA\_SIZE,
                                   8,
                                   EGL\_RENDERABLE\_TYPE,
                                   EGL\_OPENGL\_ES3\_BIT,
                                   EGL\_NONE };
        if (eglChooseConfig(eglDisplay\_, configAttribs, &config\_, 1, &count) == EGL\_FALSE) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "EglRenderContext",
                         "EglRenderContext::Init: Failed to bind choose config, error: %{public}s.", GetEglErrorString());
            return false;
        }
        const EGLint contextAttribs\[\] = { EGL\_CONTEXT\_CLIENT\_VERSION, 2, EGL\_NONE };
        eglContext\_ = eglCreateContext(eglDisplay\_, config\_, EGL\_NO\_CONTEXT, contextAttribs);
        if (eglContext\_ == EGL\_NO\_CONTEXT) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "EglRenderContext",
                         "EglRenderContext::Init: Failed to bind create context, error: %{public}s.", GetEglErrorString());
            return false;
        }
    
        OH\_LOG\_Print(LOG\_APP, LOG\_DEBUG, LOG\_PRINT\_DOMAIN, "EglRenderContext", "EglRenderContext::Init end.");
        return true;
    }
    
2.  创建NativeImage对象，然后根据OH\_NativeImage\_AcquireNativeWindow()获取NativeWindow对象，并注册NativeImage的回调方法。
    
    bool OpenGLRenderThread::CreateNativeImage()
    {
        nativeImage\_ = OH\_NativeImage\_Create(-1, GL\_TEXTURE\_EXTERNAL\_OES);
        if (nativeImage\_ == nullptr) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread", "OH\_NativeImage\_Create failed.");
            return false;
        }
        int ret = 0;
        {
            std::lock\_guard<std::mutex> lock(nativeImageSurfaceIdMutex\_);
            nativeImageWindow\_ = OH\_NativeImage\_AcquireNativeWindow(nativeImage\_);
            ret = OH\_NativeImage\_GetSurfaceId(nativeImage\_, &nativeImageSurfaceId\_);
        }
        if (ret != 0) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread",
                         "OH\_NativeImage\_GetSurfaceId failed, ret is %{public}d.", ret);
            return false;
        }
    
        nativeImageFrameAvailableListener\_.context = this;
        nativeImageFrameAvailableListener\_.onFrameAvailable = &OpenGLRenderThread::OnNativeImageFrameAvailable;
        ret = OH\_NativeImage\_SetOnFrameAvailableListener(nativeImage\_, nativeImageFrameAvailableListener\_);
        if (ret != 0) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread",
                         "OH\_NativeImage\_SetOnFrameAvailableListener failed, ret is %{public}d.", ret);
            return false;
        }
    
        return true;
    }
    
3.  在XComponent的回调函数OnSurfaceCreatedCB中，获取XComponent的NativeWindow对象。
    
    void OnSurfaceCreatedCB(OH\_NativeXComponent \*component, void \*window)
    {
        OH\_LOG\_Print(LOG\_APP, LOG\_INFO, LOG\_PRINT\_DOMAIN, "Callback", "OnSurfaceCreatedCB");
        if ((component == nullptr) || (window == nullptr)) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "Callback",
                         "onSurfaceCreatedCB: component or window is null");
            return;
        }
        char idStr\[OH\_XCOMPONENT\_ID\_LEN\_MAX + 1\] = {'\\0'};
        uint64\_t idSize = OH\_XCOMPONENT\_ID\_LEN\_MAX + 1;
        if (OH\_NativeXComponent\_GetXComponentId(component, idStr, &idSize) != OH\_NATIVEXCOMPONENT\_RESULT\_SUCCESS) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "Callback",
                         "OnSurfaceCreatedCB:Unable to get XComponent id");
            return;
        }
        std::string id(idStr);
        auto render = PluginRender::GetInstance(id);
        uint64\_t width;
        uint64\_t height;
        int32\_t xSize = OH\_NativeXComponent\_GetXComponentSize(component, window, &width, &height);
        if ((xSize != OH\_NATIVEXCOMPONENT\_RESULT\_SUCCESS) || (render == nullptr)) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "Callback",
                         "OnSurfaceCreatedCB: Unable to qet XComponent size");
            return;
        }
        render->nativeWindow = reinterpret\_cast<OHNativeWindow \*>(window);
        (void)OH\_NativeWindow\_NativeWindowHandleOpt(render->nativeWindow, SET\_BUFFER\_GEOMETRY, static\_cast<int>(width),
                                                        static\_cast<int>(height));
        if (id == "OpenGL") {
            render->openGLRenderThread\_->UpdateNativeWindow(render->nativeWindow, width, height);
        } else if(id == "Vulkan") {
            render->vulkanRenderThread\_->UpdateNativeWindow(render->nativeWindow, width, height);
        }
    }
    
    根据XComponent的NativeWindow对象，调用eglCreateWindowSurface()创建EGLSurface对象。
    
    void OpenGLRenderThread::UpdateNativeWindow(void \*window, uint64\_t width, uint64\_t height)
    {
        OH\_LOG\_Print(LOG\_APP, LOG\_DEBUG, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread", "UpdateNativeWindow.");
        auto nativeWindow = reinterpret\_cast<OHNativeWindow \*>(window);
        PostTask(\[this, nativeWindow, width, height\](OpenGLRender &renderContext) {
            if (nativeWindow\_ != nativeWindow) {
                if (nativeWindow\_ != nullptr) {
                    (void)OH\_NativeWindow\_NativeObjectUnreference(nativeWindow\_);
                }
                nativeWindow\_ = nativeWindow;
                if (nativeWindow\_ != nullptr) {
                    (void)OH\_NativeWindow\_NativeObjectReference(nativeWindow\_);
                }
    
                if (eglSurface\_ != EGL\_NO\_SURFACE) {
                    renderContext\_->DestroyEglSurface(eglSurface\_);
                    eglSurface\_ = EGL\_NO\_SURFACE;
                    CleanGLResources();
                }
            }
            if (nativeWindow\_ != nullptr) {
                (void)OH\_NativeWindow\_NativeWindowHandleOpt(nativeWindow\_, SET\_BUFFER\_GEOMETRY, static\_cast<int>(width),
                                                            static\_cast<int>(height));
                if (eglSurface\_ == EGL\_NO\_SURFACE) {
                    eglSurface\_ = renderContext\_->CreateEglSurface(static\_cast<EGLNativeWindowType>(nativeWindow\_));
                }
                if (eglSurface\_ == EGL\_NO\_SURFACE) {
                    OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread", "xfl CreateEglSurface failed.");
                    return;
                }
                renderContext\_->MakeCurrent(eglSurface\_);
                CreateGLResources();
            }
        });
    }
    
4.  初始化视频解码的环境，包括初始化解封装器、初始化解码器。注意，解码器设置的是NativeImage的NativeWindow对象。
    
    napi\_value PluginRender::StartPlayer(napi\_env env, napi\_callback\_info info)
    {
        SampleInfo sampleInfo;
        size\_t argc = 4;
        napi\_value args\[4\] = {nullptr};
        napi\_get\_cb\_info(env, info, &argc, args, nullptr, nullptr);
        napi\_get\_value\_int32(env, args\[0\], &sampleInfo.inputFd);
        napi\_get\_value\_int64(env, args\[1\], &sampleInfo.inputFileOffset);
        napi\_get\_value\_int64(env, args\[2\], &sampleInfo.inputFileSize);
        size\_t length = 0;
        napi\_status status = napi\_get\_value\_string\_utf8(env, args\[3\], nullptr, 0, &length);
        char\* buf = new char\[length + 1\];
        std::memset(buf, 0, length + 1);
        status = napi\_get\_value\_string\_utf8(env, args\[3\], buf, length + 1, &length);
        std::string type = buf;
        PluginRender \*render = PluginRender::GetInstance(type);
        if (render != nullptr) {
            if (type == "OpenGL") {
                sampleInfo.window = render->openGLRenderThread\_->GetNativeImageWindow();
            } else if (type == "Vulkan") {
                sampleInfo.window = render->vulkanRenderThread\_->GetNativeImageWindow();
            } else {
                sampleInfo.window = render->nativeWindow;
            }
        }
        int32\_t ret = Player::GetInstance().Init(sampleInfo);
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            return nullptr;
        }
        Player::GetInstance().Start();
        return nullptr;
    }
    
5.  启动解码器、解码输入子线程、解码输出子线程。（代码与使用XComponent渲染一致）
6.  通过OnNeedInputBuffer获取可用的AVBuffer后，在解码输入子线程中，将解封装器读取视频数据填充到AVBuffer中，并提交给解码器进行解码。（代码与使用XComponent渲染一致）
7.  通过OnNeedOutputBuffer获取解码的视频数据后，在解码输出子线程中，将解码后的视频数据提交给输出Surface。（代码与使用XComponent渲染一致）
8.  通过OH\_NativeImage\_AttachContext()绑定上下文，再通过OH\_NativeImage\_UpdateSurfaceImage()将视频图像缓存更新至OpenGL的纹理上。
    
    void OpenGLRenderThread::DrawImage()
    {
        if (nativeImageTexId\_ == 9999) {
            glGenTextures(1, &nativeImageTexId\_);
            glBindTexture(GL\_TEXTURE\_EXTERNAL\_OES, nativeImageTexId\_);
            // set the texture wrapping parameters
            glTexParameteri(GL\_TEXTURE\_EXTERNAL\_OES, GL\_TEXTURE\_WRAP\_S, GL\_REPEAT);
            glTexParameteri(GL\_TEXTURE\_EXTERNAL\_OES, GL\_TEXTURE\_WRAP\_T, GL\_REPEAT);
            // set texture filtering parameters
            glTexParameteri(GL\_TEXTURE\_EXTERNAL\_OES, GL\_TEXTURE\_MIN\_FILTER, GL\_LINEAR);
            glTexParameteri(GL\_TEXTURE\_EXTERNAL\_OES, GL\_TEXTURE\_MAG\_FILTER, GL\_LINEAR);
        }
        OH\_LOG\_Print(LOG\_APP, LOG\_DEBUG, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread", "DrawImage.");
    
        if (eglSurface\_ == EGL\_NO\_SURFACE) {
            OH\_LOG\_Print(LOG\_APP, LOG\_WARN, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread", "eglSurface\_ is EGL\_NO\_SURFACE");
            return;
        }
    
        OH\_NativeImage\_AttachContext(nativeImage\_, nativeImageTexId\_);
    
        int32\_t ret = OH\_NativeImage\_UpdateSurfaceImage(nativeImage\_);
        if (ret != 0) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread",
                         "OH\_NativeImage\_UpdateSurfaceImage failed, ret: %{public}d, texId: %{public}d", ret,
                         nativeImageTexId\_);
            return;
        }
        OH\_LOG\_Print(LOG\_APP, LOG\_DEBUG, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread", "OH\_NativeImage\_UpdateSurfaceImage succeed.");
    
        ret = OH\_NativeImage\_GetTransformMatrix(nativeImage\_, matrix\_);
        if (ret != 0) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "OpenGLRenderThread",
                         "OH\_NativeImage\_GetTransformMatrix failed, ret: %{public}d", ret);
            return;
        }
    
        int32\_t transformTmp = 2;
        ComputeTransformByMatrix(transformTmp, matrix\_);
    
        DrawVideoImage();
    }
    
9.  通过eglSwapBuffers将渲染的缓存提交给XComponent的NativeWindow对象进行显示。
    
    void OpenGLRender::SwapBuffers(EGLSurface surface) const
    {
        EGLBoolean ret = eglSwapBuffers(eglDisplay\_, surface);
        if (ret == EGL\_FALSE) {
            EGLint surfaceId = -1;
            eglQuerySurface(eglDisplay\_, surface, EGL\_CONFIG\_ID, &surfaceId);
            OH\_LOG\_Print(
                LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "EglRenderContext",
                "EglRenderContext::SwapBuffers: Failed to SwapBuffers on EGLSurface %{public}d, error is %{public}s.",
                surfaceId, GetEglErrorString());
        }
    }
    

#### 使用Vulkan渲染视频画面

#### \[h2\]场景描述

Vulkan是一套用来做2D和3D渲染的图形应用程序接口，能够跨平台高效访问GPU。Vulkan提供了更细粒度的硬件控制，允许开发者更精细地管理资源和状态，从而提高性能和效率。相比之下，OpenGL的一些细节对开发者是隐藏的，在部分情况下会限制性能。在实际开发实现中，Vulkan的门槛远高于OpenGL，需要手动管理内存、渲染流程等。如果开发者对性能较高，且熟悉Vulkan，可以使用Vulkan渲染视频画面。

#### \[h2\]实现原理

NativeImage是提供Surface关联OpenGL外部纹理的模块，表示图形队列的消费者端，有两种方式消费处理数据。第一种是将数据和OpenGL纹理对接，需在OpenGL环境下使用，此方式在使用OpenGL渲染视频画面中介绍过。另一种是开发者自行获取Buffer进行渲染处理，使用Vulkan渲染视频画面采用的是第二种方式。

在将NativeImage的NativeWindow对象设置给解码器后，NativeImage成为了视频解码器的数据消费者。开发者可以使用OH\_NativeImage\_AcquireNativeWindowBuffer()从NativeImage获取NativeWindowBuffer对象，消费使用OHNativeWindowBuffer对象中的视频数据。然后，通过OH\_NativeBuffer\_FromNativeWindowBuffer()方法，将OHNativeWindowBuffer对象转化成NativeBuffer。最后，将NativeBuffer提交给Vulkan，Vulkan根据NativeBuffer创建ImageView对象用于后续的渲染。同时，需要注意的是，解码器解码后的图像格式通常是YUV的格式，开发者需要对数据格式进行转化，才能正常渲染显示。

其具体实现如下所示。

1.  启动渲染子线程，加载Vulkan的动态链接库。
2.  创建NativeImage对象，并根据NativeImage获取NativeWindow对象。
3.  获取XComponent的NativeWindow对象，并根据XComponent的NativeWindow对象创建Vulkan的Surface用于显示。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/50/v3/kfrW6IhYT_aMDjCYobSuSw/zh-cn_image_0000002484494813.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=4EE8961E0581DA2898BC488C94CA6D46071DA7AE8E1B83833D001BC0BC7D328D "点击放大")
    
4.  初始化视频解码的环境，包括初始化解封装器、初始化解码器。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/6b/v3/y_8N6J-zSsuJoizwtxCpFw/note_3.0-zh-cn.png?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=13672AE6B4DCDDC2D062EBA562E13E1B2438A230903C49651FD5EACA6C655F9C)
    
    在初始化配置解码器时，与直接使用XComponent渲染的方案不同，解码器设置Surface的入参是NativeImage的NativeWindow对象，而不是XComponent的NativeWindow对象。
    
5.  启动解码器、解码输入子线程、解码输出子线程。
6.  在解码输入子线程中，通过解封装器读取视频数据，并提交给解码器。
7.  通过OnNeedOutputBuffer获取解码的视频数据后，在解码输出子线程中，将解码后的视频数据提交给输出Surface（当前Surface为NativeImage的NativeWindow对象）。
8.  在NativeImage的回调onFrameAvailable()有可用数据后，通过OH\_NativeImage\_AcquireNativeWindowBuffer()获取视频数据，并通过OH\_NativeBuffer\_FromNativeWindowBuffer()转化为NativeBuffer的类型。
9.  Vulkan根据NativeBuffer创建对应的ImageView用于采样，同时，创建对应格式的采样器，通过采样器将YUV格式的图像转化成RGBA的图像，最后，通过渲染管道进行显示。
    
    ![](https://contentcenter-vali-drcn.dbankcdn.cn/pvt_2/DeveloperAlliance_scene_100_1/85/v3/IAIyE2-mQkG82lBOEVg36w/zh-cn_image_0000002484614833.jpg?HW-CC-KV=V1&HW-CC-Date=20260612T074605Z&HW-CC-Expire=86400&HW-CC-Sign=AFCB7C7B8DE19AC98C72333D57C8B0C26BF72C7089792C51113B758744BECD71 "点击放大")
    

#### \[h2\]开发步骤

1.  启动渲染子线程，加载Vulkan的动态链接库。
    
    void VulkanRenderThread::ThreadMainLoop() {
        threadId\_ = std::this\_thread::get\_id();
        if (!InitRenderContext()) {
            return;
        }
        if (!InitNativeVsync()) {
            return;
        }
        if (!CreateNativeImage()) {
            return;
        }
        while (running\_) {
            {
                OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "RenderThread", "Waiting for Vsync.");
                std::unique\_lock<std::mutex> Lock(wakeUpMutex\_);
                wakeUpCond\_.wait(Lock, \[this\]() { return wakeup\_ || vSyncCnt\_ > 0 || availableFrameCnt\_ > 0; });
                wakeup\_ = false;
                vSyncCnt\_--;
                (void)OH\_NativeVSync\_RequestFrame(nativeVsync\_, &VulkanRenderThread::OnVsync, this);
            }
    
            std::vector<VulkanRenderTask> tasks;
            {
                std::lock\_guard<std::mutex> lock(taskMutex\_);
                tasks.swap(tasks\_);
            }
            for (const auto &task : tasks) {
                task(\*vulkanRenderContext\_);
            }
            if (availableFrameCnt\_ <= 0) {
                continue;
            }
            DrawImage();
            availableFrameCnt\_--;
        }
    }
    
    动态加载libvulkan.so，并加载Vulkan基础函数的指针。
    
    // Dynamically load Vulkan library and base function pointers
    bool LoadVulkanLibrary() {
        // Load vulkan library
        constexpr char \*path = "libvulkan.so";
        dlerror();
        g\_libVulkan = dlopen(path, RTLD\_NOW | RTLD\_LOCAL);
        if (!g\_libVulkan) {
            OH\_LOG\_ERROR(LOG\_APP, "Failed to load vulkan Library, %{public}s", dlerror());
            return false;
        }
    
        // // Load base function pointers
        vkEnumerateInstanceExtensionProperties = reinterpret\_cast<PFN\_vkEnumerateInstanceExtensionProperties>(
            dlsym(g\_libVulkan, "vkEnumerateInstanceExtensionProperties"));
        vkEnumerateInstanceLayerProperties = reinterpret\_cast<PFN\_vkEnumerateInstanceLayerProperties>(
            dlsym(g\_libVulkan, "vkEnumerateInstanceLayerProperties"));
        vkCreateInstance = reinterpret\_cast<PFN\_vkCreateInstance>(dlsym(g\_libVulkan, "vkCreateInstance"));
        vkGetInstanceProcAddr = reinterpret\_cast<PFN\_vkGetInstanceProcAddr>(dlsym(g\_libVulkan, "vkGetInstanceProcAddr"));
        vkGetDeviceProcAddr = reinterpret\_cast<PFN\_vkGetDeviceProcAddr>(dlsym(g\_libVulkan, "vkGetDeviceProcAddr"));
    
        return true;
    }
    
2.  创建NativeImage对象，并根据NativeImage获取NativeWindow对象。
    
    bool VulkanRenderThread::CreateNativeImage() {
        nativeImage\_ = OH\_NativeImage\_Create(-1, GL\_TEXTURE\_EXTERNAL\_OES);
        if (nativeImage\_ == nullptr) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "RenderThread", "OH\_NativeImage\_Create failed.");
            return false;
        }
        int ret = 0;
        {
            std::lock\_guard<std::mutex> Lock(nativeImageSurfaceIdMutex\_);
            nativeImageWindow\_ = OH\_NativeImage\_AcquireNativeWindow(nativeImage\_);
            ret = OH\_NativeImage\_GetSurfaceId(nativeImage\_, &nativeImageSurfaceId\_);
        }
        if (ret != 0) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "RenderThread",
                         "OH\_NativeImage\_GetSurfaceId failed, ret is %{public}d.", ret);
            return false;
        }
    
        nativeImageFrameAvailableListener\_.context = this;
        nativeImageFrameAvailableListener\_.onFrameAvailable = &VulkanRenderThread::OnNativeImageFrameAvailable;
        ret = OH\_NativeImage\_SetOnFrameAvailableListener(nativeImage\_, nativeImageFrameAvailableListener\_);
        if (ret != 0) {
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "RenderThread",
                         "OH\_NativeImage\_SetonFrameAvailableListener failed, ret is %{public}d.", ret);
            return false;
        }
        return true;
    }
    
3.  在获取XComponent的NativeWindow对象后，根据XComponent的NativeWindow对象创建Vulkan的Surface。
    
    void VulkanRenderThread::UpdateNativeWindow(void \*window, uint64\_t width, uint64\_t height) {
        OH\_LOG\_Print(LOG\_APP, LOG\_DEBUG, LOG\_PRINT\_DOMAIN, "RenderThread", "UpdateNativeWindow.");
        auto nativeWindow = reinterpret\_cast<OHNativeWindow \*>(window);
        PostTask(\[this, nativeWindow, width, height\](VulkanRender &renderContext) {
            if (nativeWindow\_ != nativeWindow) {
                if (nativeWindow\_ != nullptr) {
                    (void)OH\_NativeWindow\_NativeObjectUnreference(nativeWindow\_);
                }
                nativeWindow\_ = nativeWindow;
                if (nativeWindow\_ != nullptr) {
                    (void)OH\_NativeWindow\_NativeObjectReference(nativeWindow\_);
                }
            }
            if (nativeWindow\_ != nullptr) {
                vulkanRenderContext\_->SetupWindow(nativeWindow\_);
            }
        });
    }
    
    设置Vulkan的Surface，同时更新初始化Vulkan的上下文，包括Vulkan的实例、选择物理设备、加载Vulkan的方法等。
    
    void VulkanRender::SetupWindow(NativeWindow \*nativeWindow) {
        nativeWindow\_ = nativeWindow;
        CreateInstance();
        vkExample::utils::LoadVulkanFunctions(instance);
        CreateSurface();
        PickPhysicalDevice();
        CreateLogicalDevice();
        vkExample::utils::LoadVulkanFunctions(device);
    
        createSwapChain();
        createRenderPass();
        createFrameBuffersAndImages();
        createVertexBuffer();
        createUniformBuffer();
        deviceInfoInitialized = true;
    }
    
    通过vkCreateSurfaceOHOS()创建Surface对象。
    
    bool VulkanRender::CreateSurface() {
        VkSurfaceCreateInfoOHOS surfaceCreateInfo{};
        surfaceCreateInfo.sType = VK\_STRUCTURE\_TYPE\_SURFACE\_CREATE\_INFO\_OHOS;
        if (nativeWindow\_ == nullptr) {
            OH\_LOG\_INFO(LOG\_APP, "nativeWindow\_ is nulptr.Failed to create surface !");
            return false;
        }
        surfaceCreateInfo.window = nativeWindow\_;
        surfaceCreateInfo.flags = 0;
        surfaceCreateInfo.pNext = nullptr;
        bool res = CheckResult(vkCreateSurfaceOHOS(instance, &surfaceCreateInfo, nullptr, &surface));
        return res;
    }
    
4.  初始化视频解码的环境，包括初始化解封装器、初始化解码器。
    
    napi\_value PluginRender::StartPlayer(napi\_env env, napi\_callback\_info info)
    {
        SampleInfo sampleInfo;
        size\_t argc = 4;
        napi\_value args\[4\] = {nullptr};
        napi\_get\_cb\_info(env, info, &argc, args, nullptr, nullptr);
        napi\_get\_value\_int32(env, args\[0\], &sampleInfo.inputFd);
        napi\_get\_value\_int64(env, args\[1\], &sampleInfo.inputFileOffset);
        napi\_get\_value\_int64(env, args\[2\], &sampleInfo.inputFileSize);
        size\_t length = 0;
        napi\_status status = napi\_get\_value\_string\_utf8(env, args\[3\], nullptr, 0, &length);
        char\* buf = new char\[length + 1\];
        std::memset(buf, 0, length + 1);
        status = napi\_get\_value\_string\_utf8(env, args\[3\], buf, length + 1, &length);
        std::string type = buf;
        PluginRender \*render = PluginRender::GetInstance(type);
        if (render != nullptr) {
            if (type == "OpenGL") {
                sampleInfo.window = render->openGLRenderThread\_->GetNativeImageWindow();
            } else if (type == "Vulkan") {
                sampleInfo.window = render->vulkanRenderThread\_->GetNativeImageWindow();
            } else {
                sampleInfo.window = render->nativeWindow;
            }
        }
        int32\_t ret = Player::GetInstance().Init(sampleInfo);
        if (ret != AVCODEC\_SAMPLE\_ERR\_OK) {
            return nullptr;
        }
        Player::GetInstance().Start();
        return nullptr;
    }
    
5.  启动解码器、解码输入子线程、解码输出子线程。（代码与使用XComponent渲染一致）
6.  在解码输入子线程中，通过解封装器读取视频数据，并提交给解码器。（代码与使用XComponent渲染一致）
7.  在解码输出子线程中，将解码后的视频数据提交给输出Surface。（代码与使用XComponent渲染一致）
8.  在NativeImage有可用数据后，通过OH\_NativeImage\_AcquireNativeWindowBuffer()获取视频数据，并通过OH\_NativeBuffer\_FromNativeWindowBuffer()转化为NativeBuffer的类型。
    
    void VulkanRenderThread::DrawImage() {
        OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "VulkanRenderThread", "DrawImage.");
        int ret;
        OHNativeWindowBuffer \*inBuffer = nullptr;
        OH\_NativeBuffer \*nativeBuffer = nullptr;
        int32\_t fenceFd1 = -1;
        ret = OH\_NativeImage\_AcquireNativeWindowBuffer(nativeImage\_, &inBuffer, &fenceFd1);
        ret = OH\_NativeWindow\_NativeObjectReference(inBuffer);
        ret = OH\_NativeBuffer\_FromNativeWindowBuffer(inBuffer, &nativeBuffer);
        if (nativeBuffer == nullptr) {
            OH\_NativeWindow\_NativeObjectUnreference(inBuffer);
            OH\_NativeImage\_ReleaseNativeWindowBuffer(nativeImage\_, inBuffer, -1);
            OH\_LOG\_Print(LOG\_APP, LOG\_ERROR, LOG\_PRINT\_DOMAIN, "VulkanRenderThread", "nativeBuffer is null.");
            return;
        }
        ret = OH\_NativeImage\_GetTransformMatrix(nativeImage\_, matrix\_);
        int32\_t transformTmp = 0;
        ComputeTransform(transformTmp, matrix\_);
        vulkanRenderContext\_->hwBufferToTexture(nativeBuffer, matrix\_);
        vulkanRenderContext\_->render();
        if (lastInBuffer\_ != nullptr) {
            OH\_NativeWindow\_NativeObjectUnreference(lastInBuffer\_);
            OH\_NativeImage\_ReleaseNativeWindowBuffer(nativeImage\_, lastInBuffer\_, -1);
        }
        lastInBuffer\_ = inBuffer;
    }
    
9.  Vulkan根据NativeBuffer创建对应的ImageView用于渲染显示，同时，创建对应格式的采样器，将YUV格式的图像转化成RGBA的图像，用于正确的渲染显示。
    
    void VulkanRender::hwBufferToTexture(OH\_NativeBuffer \*buffer, float transformMatrix\[16\]) {
        OH\_LOG\_Print(LOG\_APP, LOG\_INFO, 0xFF00, "VulkanRenderThread", "hwBufferToTexture.");
        if (!deviceInfoInitialized) {
            return;
        }
        UniformBufferObject ubo{};
        memcpy(ubo.mvp, transformMatrix, sizeof(float) \* 16);
        memcpy(buffersInfo.uniformBufferMapped, &ubo, sizeof(ubo));
        VkNativeBufferFormatPropertiesOHOS nbFormatProps = {
            .sType = VK\_STRUCTURE\_TYPE\_NATIVE\_BUFFER\_FORMAT\_PROPERTIES\_OHOS,
            .pNext = nullptr
        };
        VkNativeBufferPropertiesOHOS nbProps = {.sType = VK\_STRUCTURE\_TYPE\_NATIVE\_BUFFER\_PROPERTIES\_OHOS,
                                                .pNext = &nbFormatProps};
        CheckResult(vkGetNativeBufferPropertiesOHOS(device, buffer, &nbProps));
    
        VkImportNativeBufferInfoOHOS importBufferInfo = {
            .sType = VK\_STRUCTURE\_TYPE\_IMPORT\_NATIVE\_BUFFER\_INFO\_OHOS,
            .pNext = nullptr,
            .buffer = buffer
        };
    
        VkMemoryDedicatedAllocateInfo dedicatedAllocateInfo = {
            .sType = VK\_STRUCTURE\_TYPE\_MEMORY\_DEDICATED\_ALLOCATE\_INFO,
            .pNext = &importBufferInfo,
            .image = VK\_NULL\_HANDLE, // wiLl be set later
            .buffer = VK\_NULL\_HANDLE
        };
    
        VkPhysicalDeviceMemoryProperties physicalDeviceMemProps;
        uint32\_t foundTypeIndex = 0;
        vkGetPhysicalDeviceMemoryProperties(gpuDevice, &physicalDeviceMemProps);
        uint32\_t memTypeCnt = physicalDeviceMemProps.memoryTypeCount;
        for (uint32\_t i = 0; i < memTypeCnt; ++i) {
            if (nbProps.memoryTypeBits & (1 << i)) {
                const VkPhysicalDeviceMemoryProperties &pdmp = physicalDeviceMemProps;
                uint32\_t supportedFLags = pdmp.memoryTypes\[i\].propertyFlags & VK\_MEMORY\_PROPERTY\_DEVICE\_LOCAL\_BIT;
                if (supportedFLags == VK\_MEMORY\_PROPERTY\_DEVICE\_LOCAL\_BIT) {
                    foundTypeIndex = i;
                    break;
                }
            }
        }
    
        VkMemoryAllocateInfo allocInfo{
            .sType = VK\_STRUCTURE\_TYPE\_MEMORY\_ALLOCATE\_INFO,
            .pNext = &dedicatedAllocateInfo,
            .allocationSize = nbProps.allocationSize,
            .memoryTypeIndex = 0 // Memory type assigned in the next step
        };
    
        mapMemoryTypeToIndex(nbProps.memoryTypeBits, VK\_MEMORY\_PROPERTY\_DEVICE\_LOCAL\_BIT, &allocInfo.memoryTypeIndex);
        VkExternalFormatOHOS externalFormat;
        externalFormat.sType = VK\_STRUCTURE\_TYPE\_EXTERNAL\_FORMAT\_OHOS;
        externalFormat.pNext = nullptr;
        externalFormat.externalFormat = 0;
        if (nbFormatProps.format == VK\_FORMAT\_UNDEFINED) {
            externalFormat.externalFormat = nbFormatProps.externalFormat;
        }
    
        VkExternalMemoryImageCreateInfo externalMemoryImageInfo = {
            .sType = VK\_STRUCTURE\_TYPE\_EXTERNAL\_MEMORY\_IMAGE\_CREATE\_INFO,
            .pNext = &externalFormat,
            .handleTypes = VK\_EXTERNAL\_MEMORY\_HANDLE\_TYPE\_OHOS\_NATIVE\_BUFFER\_BIT\_OHOS,
        };
    
        VkImageUsageFlags usageFlags = VK\_IMAGE\_USAGE\_SAMPLED\_BIT;
        if (nbFormatProps.format != VK\_FORMAT\_UNDEFINED) {
            usageFlags = usageFlags | VK\_IMAGE\_USAGE\_TRANSFER\_SRC\_BIT | VK\_IMAGE\_USAGE\_TRANSFER\_DST\_BIT |
                         VK\_IMAGE\_USAGE\_COLOR\_ATTACHMENT\_BIT | VK\_IMAGE\_USAGE\_INPUT\_ATTACHMENT\_BIT;
        }
        OH\_NativeBuffer\_Config config;
        OH\_NativeBuffer\_GetConfig(buffer, &config);
        VkImageCreateInfo image\_create\_info = {
            .sType = VK\_STRUCTURE\_TYPE\_IMAGE\_CREATE\_INFO,
            .pNext = &externalMemoryImageInfo,
            .flags = 0,
            .imageType = VK\_IMAGE\_TYPE\_2D,
            .format = nbFormatProps.format,
            .extent = {static\_cast<uint32\_t>(config.width), static\_cast<uint32\_t>(config.height), 1},
            .mipLevels = 1,
            .arrayLayers = 1,
            .samples = VK\_SAMPLE\_COUNT\_1\_BIT,
            .tiling = VK\_IMAGE\_TILING\_OPTIMAL,
            .usage = usageFlags,
            .sharingMode = VK\_SHARING\_MODE\_EXCLUSIVE,
            .queueFamilyIndexCount = 1,
            .pQueueFamilyIndices = &queueFamilyIndex\_,
            // VK\_IMAGE\_LAYOUT\_UNDEFINED is mandatory when using external memory
            .initialLayout = VK\_IMAGE\_LAYOUT\_UNDEFINED
        };
    
        CheckResult(vkCreateImage(device, &image\_create\_info, nullptr, &externalTextureImage));
        dedicatedAllocateInfo.image = externalTextureImage;
        CheckResult(vkAllocateMemory(device, &allocInfo, nullptr, &externalTextureMemory));
        CheckResult(vkBindImageMemory(device, externalTextureImage, externalTextureMemory, 0));
    
        VkSamplerYcbcrConversionCreateInfo ycbcrCreateInfo = {
            .sType = VK\_STRUCTURE\_TYPE\_SAMPLER\_YCBCR\_CONVERSION\_CREATE\_INFO,
            .ycbcrRange = nbFormatProps.suggestedYcbcrRange,
            .components = nbFormatProps.samplerYcbcrConversionComponents,
            .xChromaOffset = nbFormatProps.suggestedXChromaOffset,
            .yChromaOffset = nbFormatProps.suggestedYChromaOffset,
            .chromaFilter = VK\_FILTER\_LINEAR,
            .forceExplicitReconstruction = false
        };
    
        if (nbFormatProps.format == VK\_FORMAT\_UNDEFINED) {
            ycbcrCreateInfo.pNext = &externalFormat;
            ycbcrCreateInfo.format = VK\_FORMAT\_UNDEFINED;
            ycbcrCreateInfo.ycbcrModel = nbFormatProps.suggestedYcbcrModel;
        }
    
        CheckResult(
            vkCreateSamplerYcbcrConversion(device, &ycbcrCreateInfo, nullptr, &externalTextureInfo.ycbcrConversion));
    
        VkSamplerYcbcrConversionInfo convInfoWrap = {
            .sType = VK\_STRUCTURE\_TYPE\_SAMPLER\_YCBCR\_CONVERSION\_INFO,
            .conversion = externalTextureInfo.ycbcrConversion,
            .pNext = nullptr,
        };
    
        VkImageViewCreateInfo view = {
            .sType = VK\_STRUCTURE\_TYPE\_IMAGE\_VIEW\_CREATE\_INFO,
            .pNext = &convInfoWrap,
            .flags = 0,
            .image = externalTextureImage,
            .viewType = VK\_IMAGE\_VIEW\_TYPE\_2D,
            .format = nbFormatProps.format,
            .components =
                {
                    VK\_COMPONENT\_SWIZZLE\_IDENTITY,
                    VK\_COMPONENT\_SWIZZLE\_IDENTITY,
                    VK\_COMPONENT\_SWIZZLE\_IDENTITY,
                    VK\_COMPONENT\_SWIZZLE\_IDENTITY
                },
            .subresourceRange = {VK\_IMAGE\_ASPECT\_COLOR\_BIT, 0, 1, 0, 1},
        };
        CheckResult(vkCreateImageView(device, &view, nullptr, &externalTextureInfo.view));
    
        // Create sampler
        const VkSamplerCreateInfo sampler = {
            .sType = VK\_STRUCTURE\_TYPE\_SAMPLER\_CREATE\_INFO,
            .pNext = &convInfoWrap,
            .magFilter = VK\_FILTER\_NEAREST,
            .minFilter = VK\_FILTER\_NEAREST,
            .mipmapMode = VK\_SAMPLER\_MIPMAP\_MODE\_NEAREST,
            .addressModeU = VK\_SAMPLER\_ADDRESS\_MODE\_CLAMP\_TO\_EDGE,
            .addressModeV = VK\_SAMPLER\_ADDRESS\_MODE\_CLAMP\_TO\_EDGE,
            .addressModeW = VK\_SAMPLER\_ADDRESS\_MODE\_CLAMP\_TO\_EDGE,
            .mipLodBias = 0.0f,
            .compareEnable = VK\_FALSE,
            .anisotropyEnable = VK\_FALSE,
            .maxAnisotropy = 1,
            .compareOp = VK\_COMPARE\_OP\_NEVER,
            .minLod = 0.0f,
            .maxLod = 0.0f,
            .borderColor = VK\_BORDER\_COLOR\_FLOAT\_OPAQUE\_WHITE,
            .unnormalizedCoordinates = VK\_FALSE
        };
        CheckResult(vkCreateSampler(device, &sampler, nullptr, &externalTextureInfo.sampler));
    
        createDescriptorSetLayout();
        createDescriptorSet();
    
        VkDescriptorImageInfo imageInfo = {
            .sampler = externalTextureInfo.sampler,
            .imageView = externalTextureInfo.view,
            .imageLayout = VK\_IMAGE\_LAYOUT\_SHADER\_READ\_ONLY\_OPTIMAL
        };
    
        VkDescriptorBufferInfo bufferInfo = {
            .buffer = buffersInfo.uniformBuf, .offset = 0, .range = sizeof(UniformBufferObject)};
        VkWriteDescriptorSet bufferWrite = {.sType = VK\_STRUCTURE\_TYPE\_WRITE\_DESCRIPTOR\_SET,
                                            .dstSet = gfxPipelineInfo.descSet,
                                            .dstBinding = 0,
                                            .dstArrayElement = 0,
                                            .descriptorCount = 1,
                                            .descriptorType = VK\_DESCRIPTOR\_TYPE\_UNIFORM\_BUFFER,
                                            .pImageInfo = nullptr,
                                            .pBufferInfo = &bufferInfo,
                                            .pTexelBufferView = nullptr};
        VkWriteDescriptorSet imageWrite = {.sType = VK\_STRUCTURE\_TYPE\_WRITE\_DESCRIPTOR\_SET,
                                           .dstSet = gfxPipelineInfo.descSet,
                                           .dstBinding = 1,
                                           .dstArrayElement = 0,
                                           .descriptorCount = 1,
                                           .descriptorType = VK\_DESCRIPTOR\_TYPE\_COMBINED\_IMAGE\_SAMPLER,
                                           .pImageInfo = &imageInfo,
                                           .pBufferInfo = nullptr,
                                           .pTexelBufferView = nullptr};
        gfxPipelineInfo.descWrites\[0\] = bufferWrite;
        gfxPipelineInfo.descWrites\[1\] = imageWrite;
        vkUpdateDescriptorSets(device, 2, gfxPipelineInfo.descWrites, 0, nullptr);
    
        createGraphicsPipeline();
        createOtherStaff();
    
        recordCommandBuffer();
        OH\_LOG\_INFO(LOG\_APP, "hwBufferToTexture end");
    }
    

#### 示例代码

*   [渲染视频数据](https://gitcode.com/harmonyos_samples/video-render)