How can I manipulate a video stream in Java by treating it as a series of matrices or a data cube?

I am looking at Xuggler as a framework for decoding streaming video, but having troubles with two things:

1. Identifying a good Java library for matrix manipulation
2. Converting the video frames from Xuggler into matrices compatible with said library.

My Java is extremely rusty. I've started by adapting some Xuggler tut开发者_运维百科orials to do the job. Any recommendations for good Java Math libraries would be appreciated as well as approaches for converting BufferedImages to that library's image format.