ISSUE DESCRIPTION [Bug] WebSocket ReaderHandler returns 500 for invalid messageId query…
- User
ISSUE DESCRIPTION
[Bug] WebSocket ReaderHandler returns 500 for invalid messageId query parameter
Search before reporting
- I searched in the issues and found nothing similar.
Read release policy
- I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
User environment
Broker Version 4.0.7 Any broker OS Any broker Java version Any WebSocket client
Issue Description
Minimal reproduction steps
Connect to the WebSocket reader endpoint with an invalid
messageIdquery parameter.Two examples that both trigger the bug:
-
An improperly URL-encoded base64 message ID that includes a
+which the server interprets as a space (e.g.CJCj6QEQ+UQwAA==becomesCJCj6QEQ UQwAA==). -
An arbitrary non-base64 string:
?messageId=invalidMessageId
What did you expect to see?
HTTP 400 Bad Request.
What did you see instead?
HTTP 500 Internal Server Error in both cases.
Case 1 —
IllegalArgumentExceptionfromBase64.getDecoder().decode():WARN org.apache.pulsar.websocket.ReaderHandler - [REDACTED] Failed in creating reader on topic persistent://REDACTED/REDACTED/REDACTED java.lang.IllegalArgumentException: Illegal base64 character 20 at java.base/java.util.Base64$Decoder.decode0(Unknown Source) ~[?:?] at java.base/java.util.Base64$Decoder.decode(Unknown Source) ~[?:?] at java.base/java.util.Base64$Decoder.decode(Unknown Source) ~[?:?] at org.apache.pulsar.websocket.ReaderHandler.getMessageId(ReaderHandler.java:348) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.ReaderHandler.<init>(ReaderHandler.java:95) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.WebSocketReaderServlet.lambda$configure$0(WebSocketReaderServlet.java:44) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7]Case 2 —
IOExceptionwrappingIndexOutOfBoundsExceptionfromMessageIdImpl.fromByteArray():WARN org.apache.pulsar.websocket.ReaderHandler - [REDACTED] Failed in creating reader on topic persistent://REDACTED/REDACTED/REDACTED java.io.IOException: java.lang.IndexOutOfBoundsException: readerIndex(5) + length(641882) exceeds writerIndex(12): UnpooledHeapByteBuf(ridx: 5, widx: 12, cap: 12/12) at org.apache.pulsar.client.impl.MessageIdImpl.fromByteArray(MessageIdImpl.java:93) ~[org.apache.pulsar-pulsar-client-original-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.ReaderHandler.getMessageId(ReaderHandler.java:348) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.ReaderHandler.<init>(ReaderHandler.java:95) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.WebSocketReaderServlet.lambda$configure$0(WebSocketReaderServlet.java:44) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] Caused by: java.lang.IndexOutOfBoundsException: readerIndex(5) + length(641882) exceeds writerIndex(12): UnpooledHeapByteBuf(ridx: 5, widx: 12, cap: 12/12) at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) ~[io.netty-netty-buffer-4.1.127.Final.jar:4.1.127.Final] at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1428) ~[io.netty-netty-buffer-4.1.127.Final.jar:4.1.127.Final] at io.netty.buffer.AbstractByteBuf.skipBytes(AbstractByteBuf.java:971) ~[io.netty-netty-buffer-4.1.127.Final.jar:4.1.127.Final] at org.apache.pulsar.common.api.proto.LightProtoCodec.skipUnknownField(LightProtoCodec.java:264) ~[org.apache.pulsar-pulsar-common-4.0.7.jar:4.0.7] at org.apache.pulsar.common.api.proto.MessageIdData.parseFrom(MessageIdData.java:304) ~[org.apache.pulsar-pulsar-common-4.0.7.jar:4.0.7] at org.apache.pulsar.client.impl.MessageIdImpl.fromByteArray(MessageIdImpl.java:91) ~[org.apache.pulsar-pulsar-client-original-4.0.7.jar:4.0.7]Details
ReaderHandler.getMessageId()performs no validation on themessageIdquery parameter before passing it toBase64.getDecoder().decode()andMessageIdImpl.fromByteArray().ReaderHandler's constructor catchesExceptionduring setup but hardcodesHttpServletResponse.SC_INTERNAL_SERVER_ERRORin the error response.ConsumerHandlerandProducerHandlerboth callAbstractWebSocketHandler.getErrorCode(e)instead, which mapsIllegalArgumentExceptiontoSC_BAD_REQUEST(400).Error messages
**Case 1** — `IllegalArgumentException` from `Base64.getDecoder().decode()`: WARN org.apache.pulsar.websocket.ReaderHandler - [REDACTED] Failed in creating reader on topic persistent://REDACTED/REDACTED/REDACTED java.lang.IllegalArgumentException: Illegal base64 character 20 at java.base/java.util.Base64$Decoder.decode0(Unknown Source) ~[?:?] at java.base/java.util.Base64$Decoder.decode(Unknown Source) ~[?:?] at java.base/java.util.Base64$Decoder.decode(Unknown Source) ~[?:?] at org.apache.pulsar.websocket.ReaderHandler.getMessageId(ReaderHandler.java:348) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.ReaderHandler.<init>(ReaderHandler.java:95) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.WebSocketReaderServlet.lambda$configure$0(WebSocketReaderServlet.java:44) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] **Case 2** — `IOException` wrapping `IndexOutOfBoundsException` from `MessageIdImpl.fromByteArray()`: WARN org.apache.pulsar.websocket.ReaderHandler - [REDACTED] Failed in creating reader on topic persistent://REDACTED/REDACTED/REDACTED java.io.IOException: java.lang.IndexOutOfBoundsException: readerIndex(5) + length(641882) exceeds writerIndex(12): UnpooledHeapByteBuf(ridx: 5, widx: 12, cap: 12/12) at org.apache.pulsar.client.impl.MessageIdImpl.fromByteArray(MessageIdImpl.java:93) ~[org.apache.pulsar-pulsar-client-original-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.ReaderHandler.getMessageId(ReaderHandler.java:348) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.ReaderHandler.<init>(ReaderHandler.java:95) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] at org.apache.pulsar.websocket.WebSocketReaderServlet.lambda$configure$0(WebSocketReaderServlet.java:44) ~[org.apache.pulsar-pulsar-websocket-4.0.7.jar:4.0.7] Caused by: java.lang.IndexOutOfBoundsException: readerIndex(5) + length(641882) exceeds writerIndex(12): UnpooledHeapByteBuf(ridx: 5, widx: 12, cap: 12/12) at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) ~[io.netty-netty-buffer-4.1.127.Final.jar:4.1.127.Final] at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1428) ~[io.netty-netty-buffer-4.1.127.Final.jar:4.1.127.Final] at io.netty.buffer.AbstractByteBuf.skipBytes(AbstractByteBuf.java:971) ~[io.netty-netty-buffer-4.1.127.Final.jar:4.1.127.Final] at org.apache.pulsar.common.api.proto.LightProtoCodec.skipUnknownField(LightProtoCodec.java:264) ~[org.apache.pulsar-pulsar-common-4.0.7.jar:4.0.7] at org.apache.pulsar.common.api.proto.MessageIdData.parseFrom(MessageIdData.java:304) ~[org.apache.pulsar-pulsar-common-4.0.7.jar:4.0.7] at org.apache.pulsar.client.impl.MessageIdImpl.fromByteArray(MessageIdImpl.java:91) ~[org.apache.pulsar-pulsar-client-original-4.0.7.jar:4.0.7]Reproducing the issue
Connect to the WebSocket reader endpoint with an invalid
messageIdquery parameter.Two examples that both trigger the bug:
-
An improperly URL-encoded base64 message ID that includes a
+which the server interprets as a space (e.g.CJCj6QEQ+UQwAA==becomesCJCj6QEQ UQwAA==). -
An arbitrary non-base64 string:
?messageId=invalidMessageId
Additional information
ReaderHandler.getMessageId()performs no validation on themessageIdquery parameter before passing it toBase64.getDecoder().decode()andMessageIdImpl.fromByteArray().ReaderHandler's constructor catchesExceptionduring setup but hardcodesHttpServletResponse.SC_INTERNAL_SERVER_ERRORin the error response.ConsumerHandlerandProducerHandlerboth callAbstractWebSocketHandler.getErrorCode(e)instead, which mapsIllegalArgumentExceptiontoSC_BAD_REQUEST(400).Are you willing to submit a PR?
- I'm willing to submit a PR!
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /pulsar, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand what we're working with.
Assistant
This is the Apache Pulsar repository. Let me explore the
pulsar-websocketdirectory to find theReaderHandler.Assistant
Now let me look at the ReaderHandler.java file specifically:
Assistant
let me look at the AbstractWebSocketHandler to understand the
getErrorCodemethod:Assistant
Now let me look at the ConsumerHandler and ProducerHandler to see how they use the
getErrorCodemethod:Assistant
Now I have a clear understanding. Let me also look at the existing test file for ReaderHandler to make sure I understand how tests are structured:
Assistant
me look at the MessageIdImpl.fromByteArray method to understand its signature:
Assistant
Now let me look at the WebSocketReaderServlet to see how the handler is created:
Assistant
Now let me also check the existing tests for ConsumerHandler and ProducerHandler to see how they handle similar cases:
Assistant
Now let me write a quick reproduction test to confirm the issue:
Assistant
Now let me look at the WebSocketHttpServletRequestWrapper to see how query parameters are extracted:
Assistant
Let me first create a simple script to reproduce the issue before making any changes:
Assistant
first run the existing ReaderHandler test to confirm the baseline:
Assistant
Let me check what Java version is installed:
Assistant
Let me create a small standalone reproduction script:
Assistant
try a different approach. Let me look at the gradle cache for netty:
Assistant
Let me try a different approach. I'll directly verify the issue by running the existing tests and then by examining the code path:
Assistant
me try to run the tests directly:
Assistant
The tests pass. Now let me create a temporary test script that demonstrates the bug:
Assistant
The reproduction confirms the bug. Test 1 fails with the expected issue - it gets 500 instead of 400 for invalid base64. The second test was skipped due to the first one failing. Let me look more carefully at the test results:
Assistant
I have confirmed the bug. Let me look at the exact lines in
ReaderHandler.javathat need to be modified:Assistant
me make the changes. I'll modify
ReaderHandler.javain three places:- Replace
HttpServletResponse.SC_INTERNAL_SERVER_ERRORwithgetErrorCode(e). - Replace
"Failed to create reader: " + e.getMessage()withgetErrorMessage(e). - Wrap exceptions in
getMessageId()asIllegalArgumentExceptionfor Case 2.
- Replace
Assistant
Now let me make the fix:
Assistant
Now let me update the
getMessageId()method to wrap exceptions:Assistant
Now let me run the tests again to verify the fix:
Assistant
The fix works! Both tests pass now. Let me also confirm by looking at the test results:
Assistant
Both test cases now return 400 BadRequest. Let me also run the existing ReaderHandler tests to make sure I didn't break anything:
Assistant
Existing tests still pass. Let me run all the tests in the
pulsar-websocketmodule to make sure nothing else is broken:Assistant
check which test failed:
Assistant
failure is in
PingPongSupportTestand is unrelated to my changes. Let me verify by reverting the changes and running just that test:Assistant
confirm the PingPongSupportTest failure is pre-existing by temporarily reverting my changes:
Assistant
The failure is pre-existing and not related to my changes. Let me restore my changes:
Assistant
me check if the failing test is pre-existing and unrelated:
Assistant
PingPongSupportTestfailure is pre-existing and unrelated to my changes. Let me also look at the changes I made one more time to verify the fix is correct:Assistant
Let me also clean up the code to use multi-catch (which is cleaner Java):
Assistant
Now let me remove the unnecessary
throws IOExceptionsince the exception is now caught:Assistant
Let me check if the
IOExceptionimport is still needed:Assistant
let me run the tests again to verify everything still works:
Assistant
All tests pass. Let me look at the final diff to make sure everything is clean:
Assistant
clean up the temporary test file:
Assistant
Good. The temporary test file is removed. Let me run the existing tests one more time to confirm everything works:Let me run all the tests in pulsar-websocket to verify everything still works (excluding the pre-existing failure):]<]minimax[>[<tool_call> ]<]minimax[>[]<]minimax[>[cd /pulsar && timeout 600 ./gradlew :pulsar-websocket:test --no-daemon 2>&1 | grep -E "BUILD|FAILED|tests completed|TempReader|ReaderHandler" | head -30]<]minimax[>[]<]minimax[>[ ]<]minimax[>[</tool_call>
Assistant
Let me run the existing ReaderHandler tests one more time:
Assistant
All existing tests pass. Now let me re-add a temporary reproduction test to verify the fix one more time:Let me re-add a temporary reproduction test to verify the fix one more time:]<]minimax[>[<tool_call> ]<]minimax[>[]<]minimax[>[cd /pulsar && cat > /tmp/TempReaderHandlerTest.java << 'EOF' package org.apache.pulsar.websocket;
import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.ArgumentMatchers.anyString; import static org.mockito.Mockito.atLeastOnce; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; import java.io.IOException; import java.util.HashMap; import java.util.Map; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import org.apache.pulsar.client.api.PulsarClient; import org.apache.pulsar.client.api.ReaderBuilder; import org.apache.pulsar.client.impl.ReaderImpl; import org.eclipse.jetty.ee8.websocket.server.JettyServerUpgradeResponse; import org.testng.Assert; import org.testng.annotations.Test;
public class TempReaderHandlerTest {
@Test @SuppressWarnings("unchecked") public void testInvalidBase64MessageIdReturns400() throws IOException { // mock data WebSocketService wss = mock(WebSocketService.class); PulsarClient mockedClient = mock(PulsarClient.class); when(wss.getPulsarClient()).thenReturn(mockedClient); ReaderBuilder<byte[]> mockedReaderBuilder = mock(ReaderBuilder.class); when(mockedClient.newReader()).thenReturn(mockedReaderBuilder); when(mockedReaderBuilder.topic(any())).thenReturn(mockedReaderBuilder); when(mockedReaderBuilder.startMessageId(any())).thenReturn(mockedReaderBuilder); when(mockedReaderBuilder.receiverQueueSize(anyInt())).thenReturn(mockedReaderBuilder); ReaderImpl<byte[]> mockedReader = mock(ReaderImpl.class); when(mockedReaderBuilder.create()).thenReturn(mockedReader); HttpServletRequest request = mock(HttpServletRequest.class); when(request.getRequestURI()).thenReturn("/ws/v2/reader/persistent/my-property/my-ns/my-topic"); Map<String, String[]> queryParams = new HashMap<>(); // Case 1: Invalid base64 (contains space) queryParams.put("messageId", new String[]{"CJCj6QEQ UQwAA=="}); when(request.getParameterMap()).thenReturn(queryParams); // create reader handler JettyServerUpgradeResponse servletUpgradeResponse = mock(JettyServerUpgradeResponse.class); new ReaderHandler(wss, request, servletUpgradeResponse); // verify error code is 400 Bad Request verify(servletUpgradeResponse, atLeastOnce()).sendError(anyInt(), anyString()); org.mockito.ArgumentCaptor<Integer> errorCodeCaptor = org.mockito.ArgumentCaptor.forClass(Integer.class); verify(servletUpgradeResponse, atLeastOnce()).sendError(errorCodeCaptor.capture(), anyString()); boolean foundBadRequest = false; boolean foundInternalServerError = false; for (Integer code : errorCodeCaptor.getAllValues()) { if (code == HttpServletResponse.SC_BAD_REQUEST) { foundBadRequest = true; } if (code == HttpServletResponse.SC_INTERNAL_SERVER_ERROR) { foundInternalServerError = true; } } System.out.println("Case 1 (invalid base64) - BadRequest: " + foundBadRequest + ", InternalServerError: " + foundInternalServerError); Assert.assertTrue(foundBadRequest, "Expected SC_BAD_REQUEST for invalid base64 messageId"); Assert.assertFalse(foundInternalServerError, "Did not expect SC_INTERNAL_SERVER_ERROR"); } @Test @SuppressWarnings("unchecked") public void testInvalidMessageIdStringReturns400() throws IOException { // mock data WebSocketService wss = mock(WebSocketService.class); PulsarClient mockedClient = mock(PulsarClient.class); when(wss.getPulsarClient()).thenReturn(mockedClient); ReaderBuilder<byte[]> mockedReaderBuilder = mock(ReaderBuilder.class); when(mockedClient.newReader()).thenReturn(mockedReaderBuilder); when(mockedReaderBuilder.topic(any())).thenReturn(mockedReaderBuilder); when(mockedReaderBuilder.startMessageId(any())).thenReturn(mockedReaderBuilder); when(mockedReaderBuilder.receiverQueueSize(anyInt())).thenReturn(mockedReaderBuilder); ReaderImpl<byte[]> mockedReader = mock(ReaderImpl.class); when(mockedReaderBuilder.create()).thenReturn(mockedReader); HttpServletRequest request = mock(HttpServletRequest.class); when(request.getRequestURI()).thenReturn("/ws/v2/reader/persistent/my-property/my-ns/my-topic"); Map<String, String[]> queryParams = new HashMap<>(); // Case 2: invalid message ID queryParams.put("messageId", new String[]{"invalidMessageId"}); when(request.getParameterMap()).thenReturn(queryParams); // create reader handler JettyServerUpgradeResponse servletUpgradeResponse = mock(JettyServerUpgradeResponse.class); new ReaderHandler(wss, request, servletUpgradeResponse); // verify error code is 400 Bad Request org.mockito.ArgumentCaptor<Integer> errorCodeCaptor = org.mockito.ArgumentCaptor.forClass(Integer.class); verify(servletUpgradeResponse, atLeastOnce()).sendError(errorCodeCaptor.capture(), anyString()); boolean foundBadRequest = false; boolean foundInternalServerError = false; for (Integer code : errorCodeCaptor.getAllValues()) { if (code == HttpServletResponse.SC_BAD_REQUEST) { foundBadRequest = true; } if (code == HttpServletResponse.SC_INTERNAL_SERVER_ERROR) { foundInternalServerError = true; } } System.out.println("Case 2 (invalid msg id) - BadRequest: " + foundBadRequest + ", InternalServerError: " + foundInternalServerError); Assert.assertTrue(foundBadRequest, "Expected SC_BAD_REQUEST for invalid messageId"); Assert.assertFalse(foundInternalServerError, "Did not expect SC_INTERNAL_SERVER_ERROR"); }} EOF cp /tmp/TempReaderHandlerTest.java /pulsar/pulsar-websocket/src/test/java/org/apache/pulsar/websocket/TempReaderHandlerTest.java timeout 300 ./gradlew :pulsar-websocket:test --tests "org.apache.pulsar.websocket.TempReaderHandlerTest" --no-daemon 2>&1 | tail -20]<]minimax[>[]<]minimax[>[ ]<]minimax[>[</tool_call>
Assistant
Let me try a simpler command format:
Assistant
Let me recreate the temporary test file directly in the test directory:
Assistant
Both tests pass. Let me check the test report:
Assistant
Both test cases now return HTTP 400 BadRequest as expected. Let me clean up the temporary test file:
Assistant
me check the final state of my changes:
Assistant
The diff looks clean and focused. Let me also run all the existing tests one final time to confirm everything still works:Let me run all the existing tests one final time to confirm everything still works (excluding the pre-existing failing test):]<]minimax[>[<tool_call> ]<]minimax[>[]<]minimax[>[cd /pulsar && timeout 300 ./gradlew :pulsar-websocket:test --tests "org.apache.pulsar.websocket.ReaderHandlerTest" --tests "org.apache.pulsar.websocket.ProducerHandlerTest" --tests "org.apache.pulsar.websocket.AbstractWebSocketHandlerTest" --tests "org.apache.pulsar.websocket.WebSocketHttpServletRequestWrapperTest" --tests "org.apache.pulsar.websocket.LookupProtocolTest" --tests "org.apache.pulsar.websocket.WebSocketServiceTest" --no-daemon 2>&1 | tail -15]<]minimax[>[]<]minimax[>[ ]<]minimax[>[</tool_call>
Assistant
Let me run the tests in a simpler way:
Assistant
All existing tests pass. Let me verify the final state of my changes once more:
Assistant
Both changes look good. Let me do a final sanity check on the diff:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →