Teams会议侧边栏应用开发-会议转写

Teams应用开发，主要是权限比较麻烦，大量阅读和实践，摸索了几周，才搞明白。现将经验总结如下：

一、目标：开发一个Teams会议的侧边栏应用，实现会议的实时转写。

二、前提：

1）Teams 365基础版本以上账号Developer Portal，主要是可以登录开发者门户，建议开放管理员权限，以便可以上传开发好的APP（实际上仅仅是个mainfest.json）；

2）Teams 基础版本以上账号，可以登录Teams，添加App；

3）Azure 账号，建议开放管理员权限，以方便授予同意权限。

以上是基本要求，否则无法进行后续的工作。

三、需求分解：

1）侧边栏->关键配置（configurableTabs）

2）实时转写->转写->会议组织者ID->会议ID->用户ID

四、涉及的权限：

1）用户token=委托权限(Delegated)，如：转录，需要有会议组织者ID的用户权限；

2）应用token=应用权限(Application)，如：获取转录列表。

以上两类权限，可以通过在Azure注册应用获得，如：

五、辅助工具：

1）Graph Explorer | Try Microsoft Graph APIs - Microsoft Graph可以帮助你调试API，判断是否有权限，及需要什么权限，如：User.Read, OnlineMeetingTranscript.Read.All 等。

2）jwt.ms: Welcome!帮助你判断获得的Token是哪类Token， typ=user or app。

基本要求交代完成后，下面说说具体的，正式开始。

六、注册Azure 应用

1）先找到入口-应用注册

2）记录注册应用ID和租户ID（获取user token或 app token都要用到）

3）增加一个密钥并记录，后面就看不到了。

4）授予权限，非常重要，否则无法调用对应到接口。

5）需要设置一个回调地址（user token需要）

6）隐式获取设置（可以一步直接获取user token，存储在回调页面/auth的heders URL hash片段）

正常应该是先获取code，然后拿code换token。

以上关于注册应用的设置全部完毕。

七、应用开发

1）开发工具使用VS Code，下载Teams Tookit插件，创建一个Tab应用，使用JS语言，应用名称随意，如：MeetingRTT。

2）mainfest.json，这个很关键。注意3个地方：id(注册应用ID)、configurableTabs(侧边栏配置)和validDomains（合法域名）。

{
    "$schema": "https://developer.microsoft.com/json-schemas/teams/v1.16/MicrosoftTeams.schema.json",
    "manifestVersion": "1.16",
    "version": "1.0.0",
    "id": "ba82be1b-xxx",
    "packageName": "com.helport.transcription",
    "developer": {
        "name": "Helport",
        "websiteUrl": "https://your-website.com",
        "privacyUrl": "https://your-website.com/privacy",
        "termsOfUseUrl": "https://your-website.com/terms"
    },
    "name": {
        "short": "Meeting Transcription",
        "full": "Real-time Meeting Transcription"
    },
    "description": {
        "short": "Real-time meeting transcription",
        "full": "This app provides real-time meeting transcription in Teams meetings."
    },
    "icons": {
        "outline": "outline.png",
        "color": "color.png"
    },
    "accentColor": "#FFFFFF",
    "configurableTabs": [
        {
            "configurationUrl": "https://xxx.ngrok-free.app/config",
            "canUpdateConfiguration": true,
            "scopes": [
                "team",
                "groupchat"
            ],
            "context": [
                "meetingSidePanel",
                "meetingStage"
            ]
        }
    ],
    "permissions": [
        "identity",
        "messageTeamMembers"
    ],
    "validDomains": [
        "xxx.ngrok-free.app"
    ]
}

3) 侧边栏安装配置：一个html页面config.html

<!DOCTYPE html>
<html>
<head>
    <title>Configure Transcription</title>
    <script src="https://statics.teams.cdn.office.net/sdk/v1.11.0/js/MicrosoftTeams.min.js"></script>
</head>
<body>
    <button id="save">Save</button>
    <script>
        microsoftTeams.initialize();
        document.getElementById('save').addEventListener('click', () => {
            microsoftTeams.settings.setSettings({
                entityId: "transcriptionPanel",
                contentUrl: "https://xxx.ngrok-free.app/meetingTab",
                suggestedDisplayName: "Helport"
            });
            microsoftTeams.settings.setValidityState(true);
            microsoftTeams.settings.registerOnSaveHandler((saveEvent) => {
                saveEvent.notifySuccess();
            });
        });
    </script>
</body>
</html>

4) 本地服务端点实现，主要是为了解决跨域访问问题，需要https://，可以使用ngrok弄个免费的。

本地服务端点，主要实现有：

/confg，上面的侧边栏安装配置页面；

/meetingTab，侧边栏主页面，完成转写；

/auth，完成user token的获取，返回一个页面获取access_token(user_token)，存储在redis中；

/store_user_token，存储access_token(user_tokne)到redis中；

/get_user_token，从redis获取acces_token（user_token)，页面获取会议信息需要；

/getTranscripts，获取转录列表，需要使用app token；

/getTranscriptContent，获取转写，需要使用user_token。

import restify from "restify";
import send from "send";
import fs from "fs";
import fetch from "node-fetch";
import path from 'path';
import { fileURLToPath } from 'url';
import { storeToken, getToken } from './redisClient.js';

const __filename = fileURLToPath(import.meta.url);
console.log('__filename: ', __filename);

const __dirname = path.dirname(__filename);
console.log('__dirname: ', __dirname);

// Create HTTP server.
const server = restify.createServer({
  key: process.env.SSL_KEY_FILE ? fs.readFileSync(process.env.SSL_KEY_FILE) : undefined,
  certificate: process.env.SSL_CRT_FILE ? fs.readFileSync(process.env.SSL_CRT_FILE) : undefined,
  formatters: {
    "text/html": function (req, res, body) {
      return body;
    },
  },
});

server.use(restify.plugins.bodyParser());
server.use(restify.plugins.queryParser());

server.get(
  "/static/*",
  restify.plugins.serveStatic({
    directory: __dirname,
  })
);

server.listen(process.env.port || process.env.PORT || 3000, function () {
  console.log(`\n${server.name} listening to ${server.url}`);
});

// Adding tabs to our app. This will setup routes to various views
// Setup home page
server.get("/config", (req, res, next) => {
  send(req, __dirname + "/config/config.html").pipe(res);
});

// Setup the static tab
server.get("/meetingTab", (req, res, next) => {
  send(req, __dirname + "/panel/panel.html").pipe(res);
});

//获得用户token
server.get('/auth', (req, res, next) => {
  res.status(200);
  res.send(`
<!DOCTYPE html>
<html>
<head>
    <script>
        // Function to handle the token storage
        async function handleToken() {
            const hash = window.location.hash.substring(1);
            const hashParams = new URLSearchParams(hash);
            const access_token = hashParams.get('access_token');
            console.log('Received hash parameters:', hashParams);

            if (access_token) {
                console.log('Access token found:', access_token);
                localStorage.setItem("access_token", access_token);
                console.log('Access token stored in localStorage');
                try {
                  const response = await fetch('https://shortly-adapted-akita.ngrok-free.app/store_user_token', {
                      method: 'POST',
                      headers: {
                          'Content-Type': 'application/json'
                      },
                      body: JSON.stringify({ "user_token" : access_token })
                  });

                  if (response.ok) {
                      console.log('Token stored successfully');
                  } else {
                      console.error('Failed to store token:', response.statusText);
                  }
              } catch (error) {
                  console.error('Error storing token:', error);
              }
            } else {
                console.log('No access token found');
            }
            window.close();
        }

        // Call the function to handle the token
        handleToken();
    </script>
</head>
<body></body>
</html>
  `);
  next();
});


// 存储 user_token
server.post('/store_user_token', async (req, res) => {
  const user_token = req.body.user_token;
  if (!user_token) {
      res.status(400);
      res.send('user_token are required');
  }
  try {
      // Store user token
      await storeToken('user_token', user_token);
      console.log('user_token stored in Redis');
  } catch (err) {
      console.error('user_token store Error:', err);
  }
  res.status(200);   
  res.send('Token stored successfully');
});

// 获取 user_token
server.get('/get_user_token', async (req, res) => {
  try {
    // Store user token
    const user_token = await getToken('user_token');
    console.log('user_token get in Redis');
    res.send({"user_token": user_token});
  } catch (err) {
      console.error('user_token get Error:', err);
  }
});

//应用token
let app_token = '';

// 定义 /getTranscripts 端点
server.get('/getTranscripts', async (req, res) => {
  try {
      let token = "";
      if (app_token !=''){
        token = app_token
      }
      else{
        // 构建请求体
        const requestBody = new URLSearchParams({
          "grant_type": "client_credentials",
          "client_id": client_id, //注册应用ID
          "client_secret": client_secret, //主要应用密钥
          "scope": "https://graph.microsoft.com/.default", //默认范围，即注册应用配置的所有权限
        }).toString();

        // 获取app令牌
        const tokenUrl = `https://login.microsoftonline.com/${tenant_id}/oauth2/v2.0/token`;
        const tokenResponse = await fetch(tokenUrl, {
          method: 'POST',
          headers: {
              'Content-Type': 'application/x-www-form-urlencoded',
          },
          body: requestBody,
        });

        if (!tokenResponse.ok) {
          const errorData = await tokenResponse.json();
          res.send(500, { error: errorData.error_description });
        }
        const tokenData = await tokenResponse.json();
        app_token = tokenData.access_token
        token = app_token
        console.log("app_token recevied!")
      }

      
      const organizerId = req.query.organizerId;

      if (!organizerId) {
          res.send(400, { error: 'Organizer ID is required' });
      }

      // 调用 Microsoft Graph API
      const graphUrl = `https://graph.microsoft.com/beta/users/${organizerId}/onlineMeetings/getAllTranscripts(meetingOrganizerUserId='${organizerId}')/delta`;
      const graphResponse = await fetch(graphUrl, {
          headers: {
              Authorization: `Bearer ${token}`,
          },
      });

      if (!graphResponse.ok) {
          const errorData = await graphResponse.json();
          res.send(500, { error: errorData.error.message });
      }

      const data = await graphResponse.json();

      // 返回转录文本
      res.send(200, data);
  } catch (error) {
      // 返回错误
      res.send(500, { error: error.message });
  }
});

// 定义 /getTranscriptContent 端点
server.get('/getTranscriptContent', async (req, res) => {
  try {

    const response = await fetch('https://xxx.ngrok-free.app/get_user_token');
    const token_data = await response.json();

    const transcriptContentUrl = req.query.transcriptContentUrl;
      
      if (!transcriptContentUrl) {
          res.send(400, { error: 'transcriptContentUrl is required' });
      }
      const content_url = `${transcriptContentUrl}?$format=text/vtt`;
      console.log('content_url:', content_url)
      // 调用 Microsoft Graph API
      const graphResponse = await fetch(content_url, {
          headers: {
              Authorization: `Bearer ${token_data.user_token}`,
          },
      });

      if (!graphResponse.ok) {
          const errorData = await graphResponse.text();
          res.send(500, { error: errorData.error.message });
      }

      const data = await graphResponse.text();

      console.log('data:', data)

      // 返回转录文本
      res.send(200, data);
  } catch (error) {
      // 返回错误
      res.send(500, { error: error.message });
  }
});

5）侧边栏页面

获取当前登录用户的信息，获取会议信息，获取会议组织者ID，获取转录列表，获取转写。前提是需要获取user token才可以获取用户信息，这里使用microsoftTeams.authentication.authenticate的url去打开一个授权登录页面，由于当前用户已经登录了，默认不需要去登录，会重定向到/auth，并将user token推送到/auth页面，该/auth服务端点接收到后，存储在redis中，然后关闭页面，之所以使用redis是因为其它存储都无效，从定向后，localStorage、sessionStorage都无法保存，该页面在浏览器中和在teams中完全是两个环境，这个两个环境的数据无法交换，所以只能使用redis在服务端存储。

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Teams User Info</title>
    <script src="https://res.cdn.office.net/teams-js/2.0.0/js/MicrosoftTeams.min.js"></script>
</head>
<body>
    <button id="fetchTranscripts">Fetch Transcripts</button>
    <h2>Meeting Transcripts</h2>
    <div id="transcripts"></div>

    <script>
        const clientId ='your_client_id';
        const tenantId = 'your_tentant_id';
        const authUrl = `https://login.microsoftonline.com/${tenantId}/oauth2/v2.0/authorize`;
        const redirectUri = 'https://xxx.ngrok-free.app/auth'; // 确保与服务器端一致
        const scope = 'user.read'; //这个随便，获取到user token后会返回注册应用配置的所有权限

        

        const getUserInfo = async (accessToken) => {
            const graphUrl = 'https://graph.microsoft.com/v1.0/me';
            const response = await fetch(graphUrl, {
                headers: {
                    'Authorization': `Bearer ${accessToken}`
                }
            });

            const userInfo = await response.json();
            return userInfo;
        };

        const getMeetingDetails = async (user_token, joinMeetingId) => {
            const apiUrl = `https://graph.microsoft.com/v1.0/me/onlineMeetings?$filter=joinMeetingIdSettings/joinMeetingId eq '${joinMeetingId}'`;
            const response = await fetch(apiUrl, {
                method: 'GET',
                headers: {
                    'Authorization': `Bearer ${user_token}`,
                    'Content-Type': 'application/json'
                }
            });
        
            if (!response.ok) {
                throw new Error(`HTTP error! status: ${response.status}`);
            }
        
            const data = await response.json();
            return data.value[0];
        };

        const getTranscripts = async (organizerId) => {
            const response = await fetch(`https://xxx.ngrok-free.app/getTranscripts?organizerId=${organizerId}`);
            if (!response.ok) {
                throw new Error(`HTTP error! status: ${response.status}`);
            }

            const transcripts = await response.json();
            return transcripts;
        };

        const getTranscriptContent = async (transcriptContentUrl) => {
            const response = await fetch(`https://xxx.ngrok-free.app/getTranscriptContent?transcriptContentUrl=${transcriptContentUrl}`);
            if (!response.ok) {
                throw new Error(`HTTP error! status: ${response.status}`);
            }
        
            const content = await response.text();
            const lines = content.trim().split('\n');
            const subtitles = [];
        
            let currentSpeaker = null;
        
            for (let i = 0; i < lines.length; i++) {
                const line = lines[i].trim();
        
                if (line.includes('-->')) {
                    const [startTime, endTime] = line.split(' --> ');
                    const text = lines[i + 1].trim();
                    const speakerMatch = text.match(/<v\s*([^>]+)>/);
                    const speaker = speakerMatch ? speakerMatch[1] : null;
                    const content = text.replace(/<v\s*[^>]*>/, '').replace(/<\/v>/, '');
        
                    if (speaker && speaker !== currentSpeaker) {
                        currentSpeaker = speaker;
                    }
        
                    subtitles.push({ startTime, endTime, speaker: currentSpeaker, content });
                    i++; // Skip the next line as it's the text content
                }
            }
            return subtitles;
        };

        const displaySubtitle = (subtitle, transcriptElement) => {
            const subtitleElement = document.createElement('div');
            subtitleElement.textContent = `${subtitle.speaker}: ${subtitle.content}`;
            transcriptElement.appendChild(subtitleElement);
        };

        const displayTranscripts = (transcripts) => {
            const transcriptsContainer = document.getElementById('transcripts');
            transcriptsContainer.innerHTML = ''; // 清空之前的转录信息
        
            if (transcripts && transcripts.value && transcripts.value.length > 0) {
                transcripts.value.forEach(transcript => {
                    const transcriptElement = document.createElement('div');
                    getTranscriptContent(transcript.transcriptContentUrl)
                        .then(subtitles => {
                            subtitles.forEach(subtitle => {
                                displaySubtitle(subtitle, transcriptElement);
                            });
                        })
                        .catch(error => {
                            transcriptElement.innerHTML = `
                                <p><strong>${error}</strong></p>
                            `;
                        });
        
                    transcriptsContainer.appendChild(transcriptElement);
                });
            } else {
                transcriptsContainer.innerText = 'No transcripts found.';
            }
        };
        const init = async () => {
                microsoftTeams.app.initialize();
                microsoftTeams.authentication.authenticate({
                    url: `${authUrl}?client_id=${clientId}&response_type=token&redirect_uri=${encodeURIComponent(redirectUri)}&scope=${encodeURIComponent(scope)}`,
                    width: 600,
                    height: 535,
                    successCallback: async (result) => {
                        console.log('Authentication success:', result);
                    },
                });
                try {
                    const response = await fetch('https://shortly-adapted-akita.ngrok-free.app/get_user_token');
                    const data = await response.json();
                    if (response.ok) {
                        user_token = data.user_token;
                        console.log('user token retrieved:', user_token);
                        // Fetch user info using the access token
                        const userInfo = await getUserInfo(user_token);
                        if (userInfo)
                        {
                            console.log('User Info:', userInfo);
                        }
                        const joinMeetingId = '45756456529'; // 替换为你要查询的 joinMeetingId
                        try {
                            const meetingDetails = await getMeetingDetails(user_token, joinMeetingId);
                            console.log('Meeting Details:', meetingDetails);
                            try {
                                meetingOrganizerUserId = meetingDetails.participants.organizer.identity.user.id;
                                document.getElementById('fetchTranscripts').addEventListener('click', async () => {
                                    const organizerId = meetingOrganizerUserId
                        
                                    if (!organizerId) {
                                        console.log('Organizer ID is required');
                                        return;
                                    }
                        
                                    try {
                                        const transcripts = await getTranscripts(organizerId);
                                        console.log(transcripts)
                                        displayTranscripts(transcripts);
                                    } catch (error) {
                                        console.log(`Error: ${error.message}`);
                                    }
                                });
                            } catch (error) {
                                console.error('Error fetching transcripts:', error);
                                document.getElementById('transcripts').innerText = 'Error fetching transcripts.';
                            }
                        } catch (error) {
                            console.error('Error fetching meeting details:', error);

                        }
                            console.log('User Token:', data.user_token);
                    } else {
                        console.error('Failed to get token:', response.statusText);
                    }
                } catch (error) {
                    console.error('Error getting token:', error);
                }
        };

        init();
    </script>
</body>
</html>

到处代码就开发完毕了。

八、部署调试