Cross-modality image generation

• Main Authors: Bongio Karrman, Anton, Fan, Ryan C
• Format: Patent
• Language: English
• Published: 07.06.2022
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; IMAGE DATA PROCESSING OR GENERATION, IN GENERAL; PHYSICS

Systems, devices, methods, and computer-readable media for image processing by machine learning are provided. A method includes providing, as input to a first machine learning (ML) model trained based on image and corresponding depth data, data of a first image, the first image captured by a sensor of a first modality. The method includes receiving, from the ML model, an estimated depth per pixel of the first image. The method includes providing, as input to a second ML model trained based on the first image and a loss in constructing an estimated second image in a second modality, the first image and receiving, from the second ML model, estimated transformation parameters that transform the first image from the first modality to the second modality. The method includes using the estimated transformation parameters and the estimated depth to generate an estimated second image in the second modality.