BlenDR

My Role in BlenDR:

Along with my mentor, Jaehong Kim, we have worked on the project, BlenDR. This project aims to improve the encoding efficiency of 3D live-streaming by introducing a novel RGB-D representation and delivery method scheme. The following delineates the role I’ve taken in the project:

• 23.06~23.08 - Depth Filling: Modified existing cross-bilateral filter for our own usage (leave valid pixels, and fill in invalid pixels only)
  • Optimized the cross-bilateral filter (allowing for 30 fps live-streaming for FHD/4K videos) used for proactive depth-filling using CUDA
• 23.08~23.11 - Colorization: Improved the open-source module for video colorization (Neuricam), increasing the PSNR and SSIM values of the model’s output by approximately 15% and allowing for various size and YUV-type images as input
• 23.11~23.12 - Depth-Packing: Optimized code used for UV-channel Depth Encoding/Decoding using SIMD on a NVIDIA Jetson Nano and Xavier
• 23.12~24.02 - Full-Pipeline: Developed the full pipeline of BlenDR (depth filling, depth packing, encoding, decoding, depth unpacking, edge detection) ensuring 30 fps live-streaming capabilities
  • Use Microsoft Azure Kinect SDK to capture frames of RGB-D
  • Use FFmpeg Dev Lib to allow full end-to-end implementation
  • Mounted our implementation on the RTMP protocol on a NGINX server to simulate a real application on a network
  • Enabled real-time sender-side processing with 4K images using CUDA + (openCV GPU Module)
• 23.12~24.02 - Others:
  • Tested FFmpeg hardware/software encoding on iPhone 13 Pro to check for mobile compatibility
  • Write Implementation and Background section on the paper (other sections have also been modified and edited for readability)
  • Proof-reading

Reflection on the Experience:

I’ve wandered around quite a bit before joining this lab, trying out different areas every semester ranging from research on MEMS, epidermal sensors, and displays. I was always someone who would be scared of coding, and while this fear hasn’t completely disappeared, the Computer Networks course gave me an unfamiliar sense of interest and curiosity in the field. It was my first time in the Computer Division and in an actual research paper.

I didn’t realize that I had the capability to do this but I managed to write some lines of code that also helped out in the big picture (enable real-time capabilities, reassuring that BlenDR’s depth filling and depth packing ACTUALLY help improve the encoding process). I had the most fun when I had to deal with Jetson devices (first time dealing with embedded devices as well), since the hardware was so limited and I had to fit a heavy model into such a wimpy device — kinda tested my limits too.

I had to apply things I learned in OS and system programming courses, such as multi-threading and scheduling of threads to realize the pipelined implementation, and surprisingly the data structure class also helped come up with a neat design for our pipelined implementation. I never knew how GPU coding worked, but also got to get my hands dirty with CUDA, OpenCV, and a little bit of pytorch. The Computer Networks course also came in handy especially dealing with NGINX and RTMP (although never actually learned this in class), since I had a general idea of packet losses / TCP double ACKs etc.

I’ve always enjoyed writing, and I was glad that my writing could also help in academic writing. It was interesting to learn the tone and nuance used in research papers, and I learned that I need to work on keeping my ideas succinct and clear. I’m glad that I could get writing and Engineering together and make something out of things I enjoy doing.

Classes that helped:

• EE415: Operating Systems and System Programming for Electrical Engineering
• EE323: Computer Networks
• EE209: Programming Structure for Electrical Engineering
• CS206: Data Structure

Next Goals

• I want to take a more active role in research next time. Although I actively sought areas I could improve during the research, I felt like a kid playing in a restricted playground area, overlooked by big adults. I did what I could in the playground but I know there’s definitely more I can do outside.
• I need to read more papers/books. I was so surprised (although kinda expected this from a PhD student) by my mentor’s knowledge in the area. If I said “A”, he would know the entire alphabet. A real role model.