Monitor your WebSockets and ThreadPools for Performance Tuning via JMX

Last week, after releasing Simple RealTime and having the traffic spike from Hacker News, I needed to setup some basic monitoring for performance tuning.

Even if there’s no silver bullet in performance tuning, there are some clear points you may want to look at like: concurrent connections, buffers and thread pool executors for inbound and outbound message channels (see the configuration and performance section for further information).

The following class (based on this) exposes to JMX the number of concurrent connections and the information of the ThreadPoolTaskExecutor used for outbound messages (modify as needed, including the inbound executor for instance):

@ManagedResource
public class WebSocketMessageBrokerStatsMonitor {

	private final Map<String, WebSocketSession> webSocketSessions;
	private final ThreadPoolExecutor outboundExecutor;

	@SuppressWarnings("unchecked")
	public WebSocketMessageBrokerStatsMonitor(SubProtocolWebSocketHandler webSocketHandler, ThreadPoolTaskExecutor outboundTaskExecutor) {
		this.webSocketSessions = (Map<String, WebSocketSession>) new DirectFieldAccessor(webSocketHandler).getPropertyValue("sessions");
		this.outboundExecutor  = outboundTaskExecutor.getThreadPoolExecutor();
	}

	@ManagedAttribute
	public int getCurrentSessions() {
		return webSocketSessions.size();
	}

	@ManagedAttribute
	public String getSendBufferSize() {
		int sendBufferSize = 0;
		for (WebSocketSession session : this.webSocketSessions.values()) {
			ConcurrentWebSocketSessionDecorator concurrentSession = (ConcurrentWebSocketSessionDecorator) session;
			sendBufferSize += concurrentSession.getBufferSize();
		}
		return formatByteCount(sendBufferSize);
	}

	@ManagedAttribute
	public int getOutboundPoolSize() {
		return outboundExecutor.getPoolSize();
	}

	@ManagedAttribute
	public int getOutboundLargestPoolSize() {
		return outboundExecutor.getLargestPoolSize();
	}

	@ManagedAttribute
	public int getOutboundActiveThreads() {
		return outboundExecutor.getActiveCount();
	}

	@ManagedAttribute
	public int getOutboundQueuedTaskCount() {
		return outboundExecutor.getQueue().size();
	}

	@ManagedAttribute
	public long getOutboundCompletedTaskCount() {
		return outboundExecutor.getCompletedTaskCount();
	}

	private static String formatByteCount(long bytes) {
		int unit = 1024;
		if (bytes < unit) return bytes + " B";
		int exp = (int) (Math.log(bytes) / Math.log(unit));
		return String.format("%.1f %sB", bytes / Math.pow(unit, exp), "KMGTPE".charAt(exp - 1));
	}

}

Define the WebSocketMessageBrokerStatsMonitor as a Spring managed bean:

@Bean
public WebSocketMessageBrokerStatsMonitor statsMonitor() {
	return new WebSocketMessageBrokerStatsMonitor((SubProtocolWebSocketHandler) subProtocolWebSocketHandler(), clientOutboundChannelExecutor());
}

Enable JMX annotations to expose our Spring managed beans by registering the AnnotationMBeanExporter:

@Bean
public AnnotationMBeanExporter annotationMBeanExporter() {
    return new AnnotationMBeanExporter();
}

And you are ready to go: open jConsole / VisualVM to see your live stats (remember to add -Dcom.sun.management.jmxremote as a VM argument):

jConsole WebSockets

Track your WebSocket messages with Spring 4 and RabbitMQ

Spring 4 introduces WebSocket support with optional support for STOMP as subprotocol and transparent fallback options with SockJS.

When using STOMP over WebSockets a handy built-in message broker takes care of subscriptions and messages, everything in memory. But when it comes to scaling and clustering, going for a full-featured STOMP broker like RabbitMQ is a much better option.

The question is now: how can we track all these websocket messages?

During development is always useful to be able to see your messages. Luckily RabbitMQ has a firehose tracer and useful tracing UI plugin, an extension of the management plugin. Let’s start by enabling the plugin and tracing:

rabbitmq-plugins enable rabbitmq_tracing
rabbitmqctl trace_on [-n node] [-p vhost]

After restarting RabbitMQ, you should see a new tab called Tracing under the Admin menu.

RabbitMQ Tracing Plugin

Add a new trace, and you’ll start tracing your messages in a log file. Message format can be either text or JSON, this is an example of a text message in our log file:

==================================================================
2014-03-17 22:37:16: Message published

Node:         rabbit@localhost
Exchange:     amq.topic
Routing keys: [<<"company-1-status">>]
Properties:   [{<<"headers">>,table,
                [{<<"content-length">>,longstr,<<"37">>}]}]
Payload: 
{"username":"salmar","status":"busy"}

==================================================================
2014-03-17 22:37:16: Message received

Node:         rabbit@localhost
Exchange:     amq.topic
Queue:        amq.gen-LP763HzfjQWpRiVpcD2Tqg
Routing keys: [<<"company-1-status">>]
Properties:   [{<<"headers">>,table,
                [{<<"content-length">>,longstr,<<"37">>}]}]
Payload: 
{"username":"salmar","status":"busy"}