Internship | AI Video Encoding of 3D Human Avatars for the Metaverse

Social XR is about communication between people over distance as if you are in the same location. At TNO we use 3D video of users for this, and are working on the challenge of maximizing video quality.

At TNO, we create innovative media technologies aimed at providing people with new and rich media experiences, which they can enjoy wherever, whenever and with whomever they want. In the TNO MediaLab, we mainly develop streaming solutions on end-to-end communications, working from the capture side to the rendering side, such as coding, transport, synchronization, orchestration, digital rights management, and service delivery.

We are currently innovating and developing technical solutions to allow photo-realistic communication in eXtended Reality (XR). In this, we develop an end-to-end video pipeline for 3D video: from capture with special cameras, to processing, encoding and distribution, to rendering on AR and VR headsets. 3D video encoding can be done in various ways, and we are investigating these on performance.

A new method of encoding is by using AI, such as done by Nvidia in their Maxine project.

The goal of the assignment is to create an AI based video encoding of 3D videos of captured users, and to evaluate this by comparing it to other methods of encoding.

If you want to know more, check out some recent videos about our work:

• https://www.youtube.com/watch?v=ZNQK97d6ZB8
• https://www.youtube.com/watch?v=PcEmoM3d-dg
• https://www.youtube.com/watch?v=jGnyQBb4JXc

And check out our TNO MediaLab website, where many references, demonstrations and publications can be found.

This assignment is a combination of research and engineering. Your work will include at least:

• (Literature) investigation into 3D video and AI based encoding alternatives.
• (Literature) investigation into 3D video quality assessment methods and tools.
• Implementation of a AI-based 3D video encoding / distribution / decoding pipeline, ideally integrated in our broader platform.
• Evaluation of the implemented pipeline in comparison to existing 3D video pipeline(s) available within TNO.

Furthermore:

• Your work is part of a bigger effort, you will be working within a team of about 20 people working on various aspects of Social XR.
• The software you write may become open source so that the societal impact can be maximized.
• The demo that showcases your work will be publicly accessible.
• Depending on the results, publishing the results in a scientific paper may be possible, targeting a high-impact conference such as ACM Multimedia or Immersive Tech Week.

The expected duration of an internship project is 6-9 months including the possibility of writing your thesis and/or taking part in a scientific paper writing. Furthermore, for this work it is expected to work at our TNO location New Babylon next to The Hague Central station at least a few days a week.

Top students are invited to apply. We are looking for students with a technical background and relevant programming experience. Experience with media technology or virtual reality is a bonus. We expect a pro-active attitude, enthusiasm, curiosity and an eagerness to learn and explore.

A large part of this assignment is about software development: you need to be able to program. You should be familiar with Javascript/Typescript, Python, C#, C++, or otherwise have a formal programming education or sufficient experience with programming.

Typical expected student background:

• Computer scienc
• Software engineerin
• Telematic
• Electrical engineerin
• Media and communication

You want to work on the precursor of your career; a work placement gives you an opportunity to take a good look at your prospective future employer. TNO goes a step further. It’s not just looking that interests us; you and your knowledge are essential to our innovation. That’s why we attach a great deal of value to your personal and professional development. You will, of course, be properly supervised during your work placement and be given the scope for you to get the best out of yourself. Naturally, we provide suitable work placement compensation.

At TNO, we innovate for a healthier, safer and more sustainable life. And for a strong economy. Since 1932, we have been making knowledge and technology available for the common good. We find each other in wonder and ingenuity. We are driven to push boundaries. There is all the space and support for your talent and ambition. You work with people who will challenge you: who inspire you and want to learn from you. Our state-of-the-art facilities are there to realize your vision. What you do at TNO matters: impact makes the difference. Because with every innovation you contribute to tomorrow’s world. Read more about TNO as an employer.

At TNO we encourage an inclusive work environment, where you can be yourself. Whatever your story and whatever unique qualities you bring to the table. It is by combining our unique strengths and perspectives that we are able to develop innovations that make a real difference in society. Want to know more? Read what steps we are taking in the area of diversity and inclusion.

After the first CV selection, the application process will be conducted by the concerning department. TNO will provide a suitable internship agreement. If you have any questions about this vacancy, you can contact the contact person mentioned below.