Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0301· openaiHugging Face Suzhen/CodeChat

Java如何不使用Xuggler库将图片转换为h264的视频帧并使用websocket实时传输转换好的视频帧

  1. User

    Java如何不使用Xuggler库将图片转换为h264的视频帧并使用websocket实时传输转换好的视频帧

  2. Assistant

    可以使用JavaCV库(OpenCV+FFmpeg的Java接口),不依赖Xuggler库,实现将图片转换为h264的视频帧并通过websocket实时传输视频帧。

    步骤:

    1. 导入JavaCV库

    在Maven项目中添加以下依赖:

    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacv-platform</artifactId>
        <version>1.5.6</version>
    </dependency>
    
    1. 图片转换为视频帧

    通过JavaCV库创建FFmpegFrameRecorder并将图片转换为视频帧,具体代码如下:

    import org.bytedeco.javacv.*;
    
    public class ImageToH264 {
        // 图片文件夹路径
        private static final String imgFolder = "/path/to/img/folder";
        // 视频输出路径
        private static final String videoFile = "/path/to/output/video.mp4";
        // 视频编码器
        private static final int videoCodec = AV_CODEC_ID_H264;
        // 视频帧率
        private static final double frameRate = 5.0;
        // 视频宽度
        private static final int videoWidth = 640;
        // 视频高度
        private static final int videoHeight = 480;
    
        public static void main(String[] args) throws Exception {
            // 创建FFmpegFrameRecorder对象
            FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(videoFile, videoWidth, videoHeight);
            recorder.setVideoCodec(videoCodec);
            recorder.setFrameRate(frameRate);
            recorder.setFormat("mp4");
            recorder.start();
    
            // 指定图片文件夹路径和图片格式
            File folder = new File(imgFolder);
            File[] files = folder.listFiles((dir, name) -> name.toLowerCase(Locale.ROOT).endsWith(".jpg"));
    
            // 读取图片并转换为视频帧
            for (File file : files) {
                BufferedImage image = ImageIO.read(file);
                Java2DFrameConverter converter = new Java2DFrameConverter();
                Frame frame = converter.getFrame(image);
                recorder.record(frame);
            }
    
            // 停止录制并释放资源
            recorder.stop();
            recorder.release();
        }
    }
    

    说明:

    • 通过FFmpegFrameRecorder设置视频编码器、帧率、输出格式等参数;
    • 读取指定文件夹下图片文件,并将图片转换为视频帧;
    • FFmpegFrameRecorder录制视频帧并输出到视频文件。
    1. 通过Websocket实时传输视频帧

    使用spring-boot-starter-websocket依赖实现websocket服务端,代码如下:

    import org.bytedeco.javacv.Frame;
    import org.bytedeco.javacv.Java2DFrameConverter;
    import org.bytedeco.javacv.FFmpegFrameRecorder;
    import org.springframework.stereotype.Component;
    import javax.imageio.ImageIO;
    import java.awt.image.BufferedImage;
    import java.io.ByteArrayOutputStream;
    import java.io.File;
    import java.nio.ByteBuffer;
    import java.util.Locale;
    import java.util.concurrent.Executors;
    import java.util.concurrent.ScheduledExecutorService;
    import java.util.concurrent.TimeUnit;
    import org.springframework.stereotype.Controller;
    import org.springframework.web.bind.annotation.PathVariable;
    import org.springframework.web.bind.annotation.RequestMapping;
    import org.springframework.web.bind.annotation.ResponseBody;
    import org.springframework.web.socket.BinaryMessage;
    import org.springframework.web.socket.CloseStatus;
    import org.springframework.web.socket.TextMessage;
    import org.springframework.web.socket.WebSocketSession;
    import org.springframework.web.socket.handler.AbstractWebSocketHandler;
    
    @Component
    @Controller
    public class WebSocketHandler extends AbstractWebSocketHandler {
        private ScheduledExecutorService executor;
        private WebSocketSession session;
        private FFmpegFrameRecorder recorder;
        private Java2DFrameConverter converter;
        private String imgFolder;
        private int frameNum;
        private boolean isStopped;
    
        // 构造函数中初始化变量
        public WebSocketHandler() {
            executor = Executors.newScheduledThreadPool(1);
            recorder = new FFmpegFrameRecorder("dummy.mp4", 640, 480, 0);
            recorder.setVideoCodec(AV_CODEC_ID_H264);
            recorder.setFrameRate(5.0);
            recorder.setFormat("mp4");
            converter = new Java2DFrameConverter();
            imgFolder = "/path/to/img/folder";
            frameNum = 0;
            isStopped = false;
        }
    
        // 定时任务,每隔一段时间将新的视频帧发送给客户端
        private void sendFrame() {
            if (session != null && session.isOpen() && !isStopped) {
                try {
                    // 构建消息体,包含视频帧和帧序号
                    BufferedImage image = ImageIO.read(new File(imgFolder, String.format("%06d.jpg", frameNum)));
                    Frame frame = converter.getFrame(image);
                    ByteArrayOutputStream bos = new ByteArrayOutputStream();
                    recorder.record(frame);
                    frameNum++;
                    byte[] bytes = converter.convert(frame).clone();
                    bos.write(ByteBuffer.allocate(4).putInt(frameNum).array());
                    bos.write(bytes);
    
                    // 发送消息
                    BinaryMessage message = new BinaryMessage(bos.toByteArray());
                    session.sendMessage(message);
                } catch (Exception e) {
                    System.out.println("Sending frame failed: " + e.getMessage());
                }
            }
        }
    
        // 处理websocket连接建立事件
        @Override
        public void afterConnectionEstablished(WebSocketSession session) throws Exception {
            this.session = session;
    
            // 启动定时任务
            executor.scheduleAtFixedRate(this::sendFrame, 0, 500, TimeUnit.MILLISECONDS);
        }
    
        // 处理websocket接收消息事件
        @Override
        protected void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
            String payload = message.getPayload();
            System.out.println("Received message: " + payload);
    
            if (payload.equals("stop")) {
                // 停止定时任务
                isStopped = true;
                executor.shutdown();
                session.close();
            }
        }
    
        // 处理websocket连接关闭事件
        @Override
        public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
            System.out.println("Connection closed: " + status.getCode());
        }
    
        // 处理错误事件
        @Override
        public void handleTransportError(WebSocketSession session, Throwable exception) throws Exception {
            System.out.println("Transport error: " + exception.getMessage());
        }
    
        // 提供RESTful接口以启动websocket
        @RequestMapping("/websocket/{sessionId}")
        @ResponseBody
        public String startWebSocket(@PathVariable String sessionId) throws Exception {
            // 绑定对应的视频文件、图片文件夹等资源
            recorder.stop();
            recorder.release();
            recorder = new FFmpegFrameRecorder("/path/to/output/video.mp4", 640, 480, 0);
            recorder.setVideoCodec(AV_CODEC_ID_H264);
            recorder.setFrameRate(5.0);
            recorder.setFormat("mp4");
            recorder.start();
            imgFolder = "/path/to/img/folder";
            frameNum = 0;
            isStopped = false;
    
            // 构建websocket连接地址
            String webSocketUrl = String.format("ws://localhost:8080/ws/%s", sessionId);
    
            // 构建JS脚本,将视频帧显示在网页中
            String script = "<!DOCTYPE html>\n" +
                    "<html>\n" +
                    "<head>\n" +
                    "  <meta charset=\"UTF-8\">\n" +
                    "  <script>\n" +
                    "    var ws = new WebSocket(\"" + webSocketUrl + "\");\n" +
                    "    ws.binaryType = 'arraybuffer';\n" +
                    "    ws.onmessage = function(event) {\n" +
                    "      var byteArray = new Uint8Array(event.data);\n" +
                    "      var frameNum = byteArray[0] << 24 | byteArray[1] << 16 | byteArray[2] << 8 | byteArray[3];\n" +
                    "      var base64 = btoa(String.fromCharCode.apply(null, byteArray.slice(4)));\n" +
                    "      var img = document.getElementById(\"video\");\n" +
                    "      img.src = \"data:image/jpeg;base64,\" + base64;\n" +
                    "    };\n" +
                    "  </script>\n" +
                    "</head>\n" +
                    "<body>\n" +
                    "  <img id=\"video\" />\n" +
                    "</body>\n" +
                    "</html>\n";
    
            return script;
        }
    }
    

    说明:

    • 使用FFmpegFrameRecorder将视频帧输出到生成的视频文件;
    • 定时任务每隔一段时间读取一张图片,并将图片转换为视频帧并发送到客户端;
    • 提供/startWebSocket/{sessionId} RESTful接口以启动websocket服务。

    使用时,访问http://localhost:8080/websocket/123,即可在网页中实时播放图片生成的视频。其中,123为sessionId,可以为任意字符串。同时可以通过发送stop消息终止websocket连接。

  3. User

    以上代码需要哪些maven依赖

  4. Assistant

    以上代码需要以下Maven依赖:

    <dependencies>
        <!-- JavaCV -->
        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacv-platform</artifactId>
            <version>1.5.6</version>
        </dependency>
        
        <!-- Spring Boot -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-websocket</artifactId>
            <version>2.5.5</version>
        </dependency>
    </dependencies>
    

    其中,javacv-platform是JavaCV库的依赖,spring-boot-starter-websocket是Spring Boot的WebSocket模块的依赖。

  5. User

    Java创建websocket客户端给服务器发送数据,如何从客户端对象获取输出流

  6. Assistant

    在Java中,WebSocket客户端在发送数据时,不需要直接获取底层的输出流进行操作,而是通过WebSocketSession对象提供的sendMessage方法进行发送。

    WebSocketSession是WebSocket中的一个重要接口,表示WebSocket连接的会话,它提供了一系列方法来操作WebSocket连接和发送/接收消息等。可以通过该接口中的sendMessage()方法来实现向服务器发送消息。具体实现方式可以参考下面的代码:

    import java.io.IOException;
    import java.net.URI;
    import java.util.concurrent.CountDownLatch;
    import org.springframework.web.socket.client.standard.StandardWebSocketClient;
    import org.springframework.web.socket.WebSocketSession;
    import org.springframework.web.socket.TextMessage;
    
    public class WebSocketClient {
        private static CountDownLatch latch;
    
        public static void main(String[] args) throws Exception {
            StandardWebSocketClient client = new StandardWebSocketClient();
            TestWebSocketHandler handler = new TestWebSocketHandler();
            URI uri = new URI("ws://localhost:8080/path/to/endpoint");
            WebSocketSession session = client.doHandshake(handler, null, uri).get();
    
            // 发送消息
            String message = "Hello, WebSocket!";
            TextMessage textMessage = new TextMessage(message);
            session.sendMessage(textMessage);
    
            // 关闭会话
            session.close();
        }
    }
    
    class TestWebSocketHandler extends TextWebSocketHandler {
        @Override
        public void afterConnectionEstablished(WebSocketSession session) throws Exception {
            System.out.println("WebSocket connected");
        }
    
        @Override
        public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
            System.out.println("Received message: " + message.getPayload());
        }
    }
    

    说明:

    • 使用StandardWebSocketClient类创建WebSocket客户端。
    • 使用doHandshake()方法连接到服务器,传递TestWebSocketHandler作为WebSocket连接处理程序。
    • 在TestWebSocketHandler.afterConnectionEstablished()方法中,当WebSocket连接到服务器时输出连接成功信息。
    • 在TestWebSocketHandler.handleTextMessage()方法中,接收的数据将被打印。

    在这个例子中,通过WebSocketSession对象的sendMessage()方法发送消息,不需要显式地获取底层输出流。因此,在WebSocket编程时,无需关注底层连接的具体实现。

  7. User

    以上代码需要依赖的maven有哪些

  8. Assistant

    以上代码需要依赖以下Maven依赖:

    <dependencies>
        <!-- Spring Boot -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-dependencies</artifactId>
            <version>2.5.5</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-websocket</artifactId>
            <version>2.5.5</version>
        </dependency>
    </dependencies>
    

    其中,spring-boot-dependencies是Spring Boot的父级依赖管理器,spring-boot-starter-websocket是Spring Boot的WebSocket模块的依赖。由于使用了父级依赖管理器,这意味着您无需在程序中声明每个依赖项的版本号,因为已经被管理器在传递依赖中声明了。

  9. User

    Java使用Websocket客户端如何获取session的outputStream

  10. Assistant

    使用WebSocket客户端发送消息时,不需要直接获取Underlying Socket的OutputStream进行操作,而是通过WebSocketSession对象提供的sendMessage()方法进行发送。

    WebSocketSession是WebSocket中的一个重要接口,表示WebSocket连接的会话,它提供了一系列方法来操作WebSocket连接和发送/接收消息等。可以通过该接口中的sendMessage()方法来实现向服务器发送消息。

    如果您需要访问WebSocket连接的底层输出流OutputStream,请使用getBinaryStream()或getTextStream()方法,这将返回底层Socket通道的OutputStream。具体实现方式可以参考下面的代码:

    import java.io.IOException;
    import java.net.URI;
    import org.apache.commons.io.IOUtils;
    import org.apache.http.HttpResponse;
    import org.apache.http.client.methods.HttpPost;
    import org.apache.http.entity.StringEntity;
    import org.apache.http.impl.client.DefaultHttpClient;
    import org.springframework.web.socket.client.WebSocketClient;
    import org.springframework.web.socket.client.WebSocketConnectionManager;
    import org.springframework.web.socket.client.standard.StandardWebSocketClient;
    import org.springframework.web.socket.WebSocketSession;
    import org.springframework.web.socket.WebSocketHandler;
    import org.springframework.web.socket.TextMessage;
    
    public class WebSocketClient {
        public static void main(String[] args) throws Exception {
            String url = "ws://localhost:8080/path/to/endpoint";
            WebSocketClient webSocketClient = new StandardWebSocketClient();
            WebSocketHandler webSocketHandler = new TestWebSocketHandler();
            WebSocketSession session = webSocketClient.doHandshake(webSocketHandler, url).get();
    
            // 发送消息
            String message = "Hello, WebSocket!";
            TextMessage textMessage = new TextMessage(message);
            session.sendMessage(textMessage);
    
            // 获取底层输出流并发送消息
            session.getTextMessageHandler().setSendTimeout(5000);
            session.getTextMessageHandler().send(new MessageWriter(session.getTextMessageHandler(), "text", message));
            session.close();
        }
    }
    
    class TestWebSocketHandler implements WebSocketHandler {
        @Override
        public void afterConnectionEstablished(WebSocketSession session) throws Exception {
            System.out.println("WebSocket connected");
        }
    
        @Override
        public void handleMessage(WebSocketSession session, WebSocketMessage<?> message) throws Exception {
            System.out.println("Received message: " + message.getPayload());
        }
    }
    
    class MessageWriter implements WebSocketHandlerMessageWriter {
        private final WebSocketMessageHandler handler;
        private final String type;
        private final String message;
    
        public MessageWriter(WebSocketMessageHandler handler, String type, String message) {
            this.handler = handler;
            this.type = type;
            this.message = message;
        }
    
        @Override
        public void writeMessage() throws IOException {
            HttpResponse response = executeWebSocketRequest(handler.getUri(), type, message);
            IOUtils.closeQuietly(response.getEntity().getContent());
        }
    
        private HttpResponse executeWebSocketRequest(URI uri, String type, String message) throws IOException {
            String webSocketUrl = uri.toString().replace("http", "ws").replace("https", "wss");
            DefaultHttpClient httpClient = new DefaultHttpClient();
            HttpPost request = new HttpPost(webSocketUrl);
            String content = String.format("{\"type\":\"%s\",\"payload\":\"%s\"}", type, message);
            StringEntity params = new StringEntity(content);
            request.setEntity(params);
            request.addHeader("Connection", "Upgrade");
            request.addHeader("Upgrade", "websocket");
            HttpResponse response = httpClient.execute(request);
            return response;
        }
    }
    

    在这个例子中,通过WebSocketSession对象的sendMessage()方法发送消息。同时,如果需要访问WebSocket连接的底层输出流OutputStream,可以通过getBinaryStream()或getTextStream()方法获取底层Socket通道的OutputStream,如下所示:

    session.getBinaryMessageHandler().getSendStream();
    session.getTextMessageHandler().getSendWriter();
    

    需要注意的是,访问WebSocket连接的底层输出流OutputStream并不建议使用,因为它会绕过WebSocket的消息缓冲器。如果绕过缓冲器,可能会出现一些不可预期的结果,例如发送的多个消息可能会合并在一起并被一次性发送。推荐使用WebSocketSession对象提供的sendMessage()方法发送消息。

  11. User

    Java写一个websocket服务器接收图片的代码

  12. Assistant

    以下是一个Java编写的WebSocket服务器代码,用于接收图片信息:

    import java.io.*;
    import java.nio.ByteBuffer;
    import java.util.HashMap;
    import java.util.Map;
    import javax.websocket.*;
    import javax.websocket.server.ServerEndpoint;
    
    @ServerEndpoint(value = "/image")
    public class ImageWebSocketServer {
    
        private static Map<String, OutputStream> outputStreams = new
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.