SpringBoot对接火山引擎大模型api实现图片识别与分析

文章目录

一、前言
二、创建应用
三、后端
- 1.SDK集成
- 2.调用Rest API
四、前端

一、前言

Spring AI实战初体验——实现可切换模型AI聊天助手-CSDN博客

在这里插入图片描述

如上，在上一篇博客，我们已经实现了spring ai对接本地大模型实现了聊天机器人，但是目前有个新需求：

上传某场所的图片，通过AI进行分析，描述图片里的内容以及存在的安全隐患
进一步通过AI分析场所的安全隐患如何治理，需要依据法律法规（联网）分析

最终效果如下所示：

在这里插入图片描述

由于目前了解到的本地大模型都无法实现上述的需求，于是这次借助了火山引擎平台来实现

https://console.volcengine.com/ark/

火山引擎目前新用户会赠送每个模型50万token的体验量，对于学习、测试用还是足够的

如下所示，本次对接的模型有 doubao-vision-pro（图片识别）和 deepseek-v3（联网分析）

在这里插入图片描述

整体的逻辑：

先传入图片到doubao模型，分析图片里的场所和存在的隐患
然后将1分析的文字结果传到deepseek-v3模型联网结合法律法规分析隐患的整改措施

二、创建应用

https://console.volcengine.com/ark/

如下所示，创建2个零代码应用

在这里插入图片描述

图片识别
联网分析

三、后端

1.SDK集成

如下图所示，火山引擎里部分模型像deepseek-v3是可以直接集成SDK来对接的

在这里插入图片描述

代码示例

@RestController
@RequestMapping("/huoShan")
public class HuoShanController {
    private final ArkService service;
    private final String imageAnalyzeBotId;

    public HuoShanController(@Value("${ai.ark.apiKey}") String apiKey, @Value("${ai.ark.base-url}") String baseUrl, @Value("${ai.ark.image-analyze-botId}") String imageAnalyzeBotId) {
        this.imageAnalyzeBotId = imageAnalyzeBotId;
        this.service = ArkService.builder()
                .dispatcher(new Dispatcher())
                .connectionPool(new ConnectionPool(5, 1, TimeUnit.SECONDS))
                .baseUrl(baseUrl)
                .apiKey(apiKey)
                .build();
    }

    @PostMapping("/image/chat")
    public ResponseEntity<String> imageChat(@RequestBody String userMessage) {
        List<ChatMessage> messages = new ArrayList<>();
        messages.add(ChatMessage.builder().role(ChatMessageRole.SYSTEM).content("你是一个对中国法律法规有深入理解的专家").build());
        messages.add(ChatMessage.builder().role(ChatMessageRole.USER).content(userMessage).build());

        BotChatCompletionRequest chatCompletionRequest = BotChatCompletionRequest.builder()
                .botId(imageAnalyzeBotId)
                .messages(messages)
                .build();

        BotChatCompletionResult chatCompletionResult = service.createBotChatCompletion(chatCompletionRequest);
        StringBuilder result = new StringBuilder();
        chatCompletionResult.getChoices().forEach(choice -> result.append(choice.getMessage().getContent()));

        return ResponseEntity.ok(result.toString());
    }
}

相关的apiKey、base-url、botId都可以从火山的API调用指南获取，获取完我们配置在application.yml里就可以从上面的代码获取

2.调用Rest API

有些模型例如doubao-pro-vision就没提供java SDK，所以需要采用直接调用rest api的方式来对接

在这里插入图片描述

代码示例

    @PostMapping("/notStream")
    public Mono<String> imageAnalysis(MultipartFile file) {
        return imageAiService.imageAnalysisNotStream(file);
    }

@Service
@Slf4j
public class ImageAiService {

    @Value("${ai.ark.image-botId}")
    private String MODEL;

    private final WebClient webClient;

    public ImageAiService(@Value("${ai.ark.apiKey}") String apiKey, @Value("${ai.ark.base-url}") String baseUrl) {
        this.webClient = WebClient.builder()
                .clientConnector(new ReactorClientHttpConnector(HttpClient.create()))
                .baseUrl(baseUrl)
                .defaultHeader("Authorization", "Bearer "+ apiKey)
                .build();
        
    public Mono<String> imageAnalysisNotStream(MultipartFile file) {
        // 压缩图片并转成 Base64 格式
        String base64 = ImageCompressor.compressImageFileToBase64UnderSize(file, 400, 400, 100);
        if (base64 == null || base64.isEmpty()) {
            log.error("图片压缩失败");
            return Mono.just("图片压缩失败");
        }

        // 构造请求体
        Map<String, Object> body = new HashMap<>();
        body.put("model", MODEL);
        // 非流式返回
        body.put("stream", false);
        body.put("stream_options", Map.of("include_usage", true));

        Map<String, Object> imageContent = Map.of(
                "type", "image_url",
                "image_url", Map.of("url", base64)
        );

        Map<String, Object> message = Map.of(
                "role", "user",
                "content", List.of(imageContent)
        );
        body.put("messages", List.of(message));

        // 调用非流式接口，直接返回拼接后的完整结果字符串
        return webClient.post()
                .uri("/bots/chat/completions")
                .contentType(MediaType.APPLICATION_JSON)
                .bodyValue(body)
                // 此时接口返回的是 JSON 数据，所以指定 JSON 类型
                .accept(MediaType.APPLICATION_JSON)
                .retrieve()
                .bodyToMono(String.class)
                .map(responseStr -> {
                    try {
                        // 解析返回结果，取出 assistant 返回的内容
                        JsonNode jsonNode = new ObjectMapper().readTree(responseStr);
                        // 此处根据实际返回结构调整解析逻辑
                        JsonNode contentNode = jsonNode.path("choices")
                                .get(0)
                                .path("message")
                                .path("content");
                        return contentNode.asText();
                    } catch (Exception e) {
                        log.error("解析返回结果异常", e);
                        return "解析返回结果异常";
                    }
                });
    }    
        
    }

注意：

上述调用火山引擎api都是非流式的，如果流式输出就把stream设置成true，再使用Flux类或SseEmitter类去接收返回就行，但是由于我流式输出得到的结果前端进行格式处理时候总是有问题，所以改用了非流式，等完整答案出来后再一次性处理格式化

/**
 * 压缩到不超过100KB的Base64编码
 */
public static String compressImageFileToBase64UnderSize(MultipartFile file, int maxWidth, int maxHeight, int maxSizeKB) {
    try {
        // 读取 MultipartFile 图片
        BufferedImage originalImage = ImageIO.read(file.getInputStream());

        // 按比例缩放图片
        Image scaledImage = originalImage.getScaledInstance(maxWidth, maxHeight, Image.SCALE_SMOOTH);
        BufferedImage resizedImage = new BufferedImage(maxWidth, maxHeight, BufferedImage.TYPE_INT_RGB);
        Graphics2D g2d = resizedImage.createGraphics();
        g2d.drawImage(scaledImage, 0, 0, null);
        g2d.dispose();

        // 获取JPEG写入器
        Iterator<ImageWriter> writers = ImageIO.getImageWritersByFormatName("jpg");
        if (!writers.hasNext()) throw new IllegalStateException("No writers found for jpg");
        ImageWriter writer = writers.next();

        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        MemoryCacheImageOutputStream output = new MemoryCacheImageOutputStream(baos);
        writer.setOutput(output);

        // 设置初始压缩质量
        float quality = 1.0f;
        byte[] imageBytes;

        do {
            baos.reset();
            ImageWriteParam param = writer.getDefaultWriteParam();
            param.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
            param.setCompressionQuality(quality);

            writer.write(null, new IIOImage(resizedImage, null, null), param);
            output.flush();
            imageBytes = baos.toByteArray();

            quality -= 0.05f; // 每次降低压缩质量
        } while (imageBytes.length > maxSizeKB * 1024 && quality > 0.05f);

        writer.dispose();
        output.close();

        //System.out.println("Final image size: " + (imageBytes.length / 1024) + " KB, final quality: " + quality);

        return "data:image/jpeg;base64," + Base64.getEncoder().encodeToString(imageBytes);

    } catch (IOException e) {
        e.printStackTrace();
        return null;
    }
}

注意：

Map<String, Object> imageContent = Map.of(
        "type", "image_url",
        "image_url", Map.of("url", base64)
);

这里的图片可以传递http/https网络地址或者图片的base64编码，由于我想是用电脑本地的文件来测试，所以采用图片转base64编码的方式来传递

四、前端

基于上次的html，新增了图片上传，图片回显，调用新接口等处理

直接放完整代码：

<!DOCTYPE html>
<html lang="zh">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>AI 聊天</title>
    <style>
        html, body {
            height: 100%;
            width: 100%;
            margin: 0;
            background-color: #f9f9f9;
            display: flex;
            align-items: center;
            justify-content: center;
        }

        .container {
            display: flex;
            flex-direction: column;
            height: 90vh;
            max-width: 800px;
            width: 100%;
            margin: auto;
        }

        .chat-container {
            flex: 1;
            display: flex;
            flex-direction: column;
            background: white;
            border-radius: 10px;
            padding: 20px;
            box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
            overflow-y: auto; /* 确保内容超出时显示滚动条 */
            min-height: 0; /* 防止 flex 容器压缩子元素 */
        }

        .chat-container::-webkit-scrollbar {
            width: 8px;
        }

        .chat-container::-webkit-scrollbar-track {
            background: #f1f1f1;
            border-radius: 4px;
        }

        .chat-container::-webkit-scrollbar-thumb {
            background: #888;
            border-radius: 4px;
        }

        .chat-container::-webkit-scrollbar-thumb:hover {
            background: #555;
        }
        .ai-message h3 {
            font-size: 1.2em;
            margin-top: 1em;
        }

        .ai-message ul {
            padding-left: 1.5em;
        }

        .ai-message li {
            margin-bottom: 0.5em;
        }

        .ai-message {
            white-space: pre-wrap;
        }
        .loading-spinner {
            border: 4px solid #f3f3f3;
            border-top: 4px solid #007bff;
            border-radius: 50%;
            width: 30px;
            height: 30px;
            animation: spin 1s linear infinite;
            margin: 10px auto;
        }

        @keyframes spin {
            0% { transform: rotate(0deg); }
            100% { transform: rotate(360deg); }
        }
        .message {
            padding: 10px 15px;
            border-radius: 15px;
            margin: 5px 0;
            max-width: 80%;
            word-wrap: break-word;
        }

        .user-message {
            background-color: #007bff;
            color: white;
            align-self: flex-end;
        }

        .ai-message {
            background-color: #e5e5e5;
            color: black;
            align-self: flex-start;
        }

        .think-message {
            background-color: #add8e6;
            color: black;
            border-radius: 10px;
            padding: 10px;
            margin: 5px 0;
            max-width: 80%;
            align-self: flex-start;
            font-style: italic;
        }

        .think-content {
            flex: 1; /* 允许内容自由扩展 */
            overflow-y: auto; /* 内容过多时显示滚动条 */
            padding: 5px;
        }

        .think-title {
            font-weight: bold;
            margin-bottom: 5px;
            display: flex;
            align-items: center;
        }

        .toggle-button {
            padding: 5px 10px;
            background-color: #007bff;
            color: white;
            border: none;
            border-radius: 5px;
            cursor: pointer;
            margin-right: 10px;
        }

        .toggle-button:hover {
            background-color: #0056b3;
        }

        .input-container {
            display: flex;
            flex-direction: column;
            padding: 10px;
            background: white;
            box-shadow: 0 -2px 5px rgba(0, 0, 0, 0.1);
        }

        .model-container {
            display: flex;
            align-items: center;
            margin-bottom: 5px;
        }

        .model-label {
            margin-right: 10px;
            font-weight: bold;
        }

        .model-select {
            padding: 5px;
            border-radius: 5px;
            border: 1px solid #ccc;
        }

        .input-box-container {
            display: flex;
            align-items: center;
        }

        .input-box {
            flex: 1;
            padding: 10px;
            border: 1px solid #ccc;
            border-radius: 5px;
        }

        .send-button, .clear-button, .stop-button {
            padding: 10px 20px;
            margin-left: 10px;
            color: white;
            border: none;
            border-radius: 5px;
            cursor: pointer;
        }

        .send-button { background-color: #007bff; }
        .send-button:hover { background-color: #0056b3; }
        .send-button:disabled { background-color: #a0c4ff; cursor: not-allowed; }

        .clear-button { background-color: #dc3545; }
        .clear-button:hover { background-color: #a71d2a; }
        .clear-button:disabled { background-color: #f5a6a6; cursor: not-allowed; }

        .stop-button { background-color: #ff9800; }
        .stop-button:hover { background-color: #e68900; }
        .stop-button:disabled { background-color: #ffb74d; cursor: not-allowed; }
    </style>
    <script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
</head>
<body>
<div class="container">
    <div class="chat-container" id="chatContainer">
        <div class="message ai-message">👋 你好，我是你的 AI 助手！</div>
    </div>
    <div class="input-container">
        <div class="model-container">
            <span class="model-label">选择模型:</span>
            <select id="modelSelect" class="model-select" onchange="changeModel()">
                <option value="deepseek-r1:latest">DeepSeek-R1（推理）</option>
                <option value="qwen:7b">Qwen</option>
                <option value="image-analysis">火山引擎-Doubao（场所图片分析，无记忆）</option>
            </select>
        </div>
        <div class="input-box-container">
            <input id="userInput" class="input-box" placeholder="请输入消息...">
            <input id="imageUpload" type="file" accept="image/jpeg, image/png" style="display: none;" onchange="validateFile()" />
            <button id="sendButton" class="send-button" onclick="handleSend()">发送</button>
            <button id="clearButton" class="clear-button" onclick="clearMemory()">清除上下文</button>
            <button id="stopButton" class="stop-button" onclick="stopAIResponse()">停止回答</button>
        </div>
    </div>
</div>
<script>
    const chatContainer = document.getElementById('chatContainer');
    const userInput = document.getElementById('userInput');
    const modelSelect = document.getElementById('modelSelect');
    const sendButton = document.getElementById('sendButton');
    const clearButton = document.getElementById('clearButton');
    const stopButton = document.getElementById('stopButton');
    let userId = '1';
    let currentModel = modelSelect.value;
    let eventSource = null;

    function validateFile() {
        const fileInput = document.getElementById('imageUpload');
        const file = fileInput.files[0];

        if (file) {
            const fileSize = file.size / 1024; // 文件大小，单位为KB
            const fileType = file.type.toLowerCase();

            // 判断文件大小是否小于200KB，格式是否为JPG或PNG
            if (fileSize > 200) {
                alert('文件大小必须小于200KB。');
                fileInput.value = ''; // 清空选择框
                return;
            }

            if (fileType !== 'image/jpeg' && fileType !== 'image/png') {
                alert('仅允许上传JPG和PNG格式的图片。');
                fileInput.value = ''; // 清空选择框
                return;
            }

        }
    }
    function handleSend() {
        if (currentModel === 'image-analysis') {
            const fileInput = document.getElementById('imageUpload');
            const file = fileInput.files[0];
            if (!file) {
                alert("请选择图片文件");
                return;
            }
            uploadAndAnalyzeImage(file);
        } else {
            sendMessage();
        }
    }

    function uploadAndAnalyzeImage(file) {
        const formData = new FormData();
        formData.append('file', file);

        // 回显图片
        const reader = new FileReader();
        reader.onload = function(e) {
            const imgElement = document.createElement('img');
            imgElement.src = e.target.result;
            imgElement.style.maxWidth = '200px';
            imgElement.style.borderRadius = '10px';
            imgElement.style.margin = '10px 0';

            const userImgMessage = document.createElement('div');
            userImgMessage.classList.add('message', 'user-message');
            userImgMessage.appendChild(imgElement);
            chatContainer.appendChild(userImgMessage);

            toggleAllButtons(false);           // 禁用按钮
            showLoadingSpinner();          // 显示加载动画

            chatContainer.scrollTop = chatContainer.scrollHeight;
        };
        reader.readAsDataURL(file);



        fetch('http://192.168.100.72:8081/image/notStream', {
            method: 'POST',
            body: formData
        })
            .then(response => response.text())
            .then(text => {
                const fixedText = text.replace(/\\n/g, '\n');
                const html = marked.parse(fixedText);

                const aiMessage = document.createElement('div');
                aiMessage.classList.add('message', 'ai-message');
                aiMessage.innerHTML = html;
                chatContainer.appendChild(aiMessage);

                // 添加进一步分析提示与按钮
                const followUp = document.createElement('div');
                followUp.classList.add('message', 'ai-message');
                followUp.innerHTML = `
    <div style="display: flex; align-items: center;">
        <span style="margin-right: 10px;">是否进一步分析风险隐患及对应整改措施？</span>
        <button class="send-button" onclick="startRiskAnalysis(\`${text.replace(/`/g, '\\`')}\`)">是</button>
    </div>
`;
                chatContainer.appendChild(followUp);
            })
            .catch(err => {
                console.error("图片分析请求失败:", err);
                appendMessage("❌ 图片分析失败", "ai-message");
            })
            .finally(() => {
                hideLoadingSpinner();     // 移除加载动画
                toggleAllButtons(true);      // 启用按钮
                chatContainer.scrollTop = chatContainer.scrollHeight;
            });
    }

    function startRiskAnalysis(content) {
        toggleAllButtons(false); // 禁用按钮

        // 显示转圈动画
        showLoadingSpinner();

        fetch('http://192.168.100.72:8081/huoShan/image/chat', {
            method: 'POST',
            headers: {
                'Content-Type': 'text/plain'
            },
            body: content
        })
            .then(response => response.text())  // 获取文本响应
            .then(result => {
                // 隐藏转圈动画
                hideLoadingSpinner()

                // 格式化结果并解析为 HTML
                const fixedText = result.replace(/\\n/g, '\n');  // 解除转义
                const html = marked.parse(fixedText);

                // 创建新的 AI 消息并添加到界面
                const aiMessage = document.createElement('div');
                aiMessage.classList.add('message', 'ai-message');
                aiMessage.innerHTML = html;
                chatContainer.appendChild(aiMessage);

                chatContainer.scrollTop = chatContainer.scrollHeight;  // 滚动到底部
                toggleAllButtons(true); // 启用按钮
            })
            .catch(err => {
                console.error("风险分析请求失败:", err);

                // 隐藏转圈动画并显示失败消息
                hideLoadingSpinner()
                appendMessage("❌ 风险分析请求失败", "ai-message");

                toggleAllButtons(true); // 启用按钮
            });
    }




    function showLoadingSpinner() {
        const spinner = document.createElement('div');
        spinner.id = 'loadingSpinner';
        spinner.className = 'loading-spinner';
        chatContainer.appendChild(spinner);
        chatContainer.scrollTop = chatContainer.scrollHeight;
    }

    function hideLoadingSpinner() {
        const spinner = document.getElementById('loadingSpinner');
        if (spinner) spinner.remove();
    }

    function sendMessage() {
        let message = userInput.value.trim();
        if (!message) return;
        appendMessage(message, 'user-message');
        streamAIResponse(userId, message);
        userInput.value = '';
    }

    function appendMessage(text, type) {
        const messageElement = document.createElement('div');
        messageElement.classList.add('message', type);
        messageElement.textContent = text;
        chatContainer.appendChild(messageElement);
        chatContainer.scrollTop = chatContainer.scrollHeight;
    }

    function streamAIResponse(userId, message) {
        // 先终止可能存在的旧 eventSource
        if (eventSource) {
            eventSource.close();
        }

        eventSource = new EventSource(`http://192.168.100.72:8081/ai/chatStreamWithMemory?userId=${encodeURIComponent(userId)}&message=${encodeURIComponent(message)}&model=${encodeURIComponent(currentModel)}`);

        let aiMessage = null;
        let thinkMode = false;
        let thinkMessage = null;

        eventSource.onmessage = event => {
            let response = event.data;

            if (response.includes('<think>') && currentModel === 'deepseek-r1:latest') {
                thinkMode = true;
                response = response.replace('<think>', '');

                // 创建思考过程气泡
                thinkMessage = document.createElement('div');
                thinkMessage.classList.add('think-message');
                thinkMessage.innerHTML = `
                <div class="think-title">
                    <button class="toggle-button" onclick="toggleThinkMessage(this)">折叠</button>
                    <span class="think-title-text">思考过程：</span>
                </div>
                <div class="think-content" style="display: block;"></div>
            `;
                chatContainer.appendChild(thinkMessage);
            }

            if (thinkMode) {
                const thinkContent = thinkMessage.querySelector('.think-content');
                if (response.includes('</think>')) {
                    response = response.replace('</think>', '');
                    thinkMode = false;
                    aiMessage = document.createElement('div');
                    aiMessage.classList.add('message', 'ai-message');
                    chatContainer.appendChild(aiMessage);
                }
                thinkContent.innerHTML += response;
            } else {
                if (!aiMessage) {
                    aiMessage = document.createElement('div');
                    aiMessage.classList.add('message', 'ai-message');
                    chatContainer.appendChild(aiMessage);
                }
                aiMessage.textContent += response;
            }

            chatContainer.scrollTop = chatContainer.scrollHeight;
        };

        eventSource.onerror = () => {
            eventSource.close();
            toggleButtons(true);
        };

        eventSource.onopen = () => {
            toggleButtons(false);
        };

        eventSource.addEventListener("close", () => {
            toggleButtons(true);
        });
    }
    function toggleAllButtons(enabled) {
        sendButton.disabled = !enabled;
        clearButton.disabled = !enabled;
        stopButton.disabled = !enabled;
    }
    function toggleButtons(enabled) {
        sendButton.disabled = !enabled;
        clearButton.disabled = !enabled;
    }
    function toggleThinkMessage(button) {
        const thinkMessage = button.closest('.think-message');
        const thinkContent = thinkMessage.querySelector('.think-content');

        if (thinkContent.style.display === 'none') {
            thinkContent.style.display = 'block';
            button.textContent = '折叠';
        } else {
            thinkContent.style.display = 'none';
            button.textContent = '展开';
        }
    }

    function stopAIResponse() {
        if (eventSource) {
            eventSource.close();
            eventSource = null;
        }

        fetch(`http://192.168.100.72:8081/ai/stopChat?userId=${userId}`, { method: 'GET' })
            .then(() => appendMessage('AI 回答已停止。', 'ai-message'))
            .catch(error => console.error(error));
    }

    function clearMemory() {
        fetch(`http://192.168.100.72:8081/ai/clearMemory?userId=${userId}`, { method: 'GET' })
            .then(() => appendMessage('上下文已清除。', 'ai-message'))
            .catch(error => console.error(error));
    }

    function changeModel() {
        currentModel = modelSelect.value;
        let currentModelName = modelSelect.options[modelSelect.selectedIndex].text;
        appendMessage(`已切换模型为 ${currentModelName}`, 'ai-message');

        if (currentModel === 'image-analysis') {
            userInput.disabled = true;
            userInput.placeholder = '请选择图片进行分析...';
            sendButton.textContent = '分析图片';
            document.getElementById('imageUpload').style.display = 'block';
        } else {
            userInput.disabled = false;
            userInput.placeholder = '请输入消息...';
            sendButton.textContent = '发送';
            document.getElementById('imageUpload').style.display = 'none';
        }
    }

    userInput.addEventListener('keypress', event => {
        if (event.key === 'Enter') {
            sendMessage();
        }
    });
</script>
</body>
</html>