DFG FOR “Learning to Sense” – Repositories

SynthEllips

https://github.com/Muhammad-Kazim/qpi-deep-cwfs

SynthEllips is a coded wavefront sensing-based synthetic dataset consisting of 20.000 datapoints created using this repository. For each datapoint, a refractive index volume composed of a random configuration of ellipsoids (refractive indices, positions, diameters, and rotations) is imaged using a wave-optical simulation of the Coded Wavefront Sensing pipeline to create a reference-specimen speckle image pair. The amplitude and scaled gradient vector field in the phase mask are also provided to assist with the supervised training of optical flow neural networks to estimate the quantitative phase from a reference-specimen speckle image pair. Training with SynthEllips demonstrated strong generalization to real biological specimens recorded experimentally, and to optical systems with different diffusers/phase masks and microscopes. The QPI performance of these networks is found to be quantitatively and qualitatively superior to classical phase retrieval methods.

Synthetic Dataset:
https://doi.org/10.5281/zenodo.18983874

Further Reading:
Syed Muhammad Kazim, Patrick Mueller, Andrii Nehrych, Ivo Ihrke: Quantitative phase imaging with deep1 coded wavefront sensing. Preprint, Optica Open 2026.

RobustSpring

https://spring-benchmark.org/

Learning-based vision systems for scene understanding must not only be accurate, but also robust to degradations that arise in real-world sensing conditions, such as blur, noise, compression artifacts, and adverse weather. To support research in this direction, we introduce RobustSpring, a benchmark dataset for evaluating robustness to image corruptions in optical flow, scene flow, and stereo. Based on the Spring benchmark, RobustSpring provides 20 corrupted versions of stereo video data, with corruptions integrated consistently in time, stereo, and depth where applicable. In total, the dataset contains 40,000 frames, or 20,000 stereo frame pairs, and enables standardized robustness evaluation alongside accuracy on the same benchmark. This makes it possible to systematically study the stability and real-world applicability of dense matching models under challenging visual conditions.

Dataset:
https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/DARUS-5047

Further reading:
Victor Oei*, Jenny Schmalfuss*, Lukas Mehl, Madlen Bartsch, Shashank Agnihotri, Margret Keuper, Andreas Bulling, Andres Bruhn: RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo. In The Fourteenth International Conference on Learning Representations (ICLR) (2026).

GeoDiv

https://abhipsabasu.github.io/geodiv/samples.html

Learning-based vision systems are increasingly used to generate visual content, yet their outputs often lack geographical diversity and can reinforce region-specific socio-economic stereotypes. In the spirit of Learning to Sense, this raises the broader question of how well modern generative models capture the diversity of the visual world across different countries and contexts, and how such behavior can be measured systematically. To support research in this direction, we introduce GeoDiv, a framework and dataset for measuring geographical diversity in text-to-image models. GeoDiv comprises 160,000 synthetic images generated by four open-source diffusion models across 10 common entities and 16 countries, together with structured annotations and interpretable diversity scores along socio-economic and visual dimensions. This enables systematic analysis of geographical bias and diversity in generative vision systems, and supports the study of how learning-based models represent the world beyond conventional performance metrics.

Dataset:
https://drive.google.com/drive/folders/1QtXcxCzPq8iteq1FehFDjNmLMKZhaLUE

Further reading:
Abhipsa Basu*, Mohana Singh*, Shashank Agnihotri, Margret Keuper, R. Venkatesh Babu: GeoDiv: Framework for Measuring Geographical Diversity in Text-to-Image Models. In The Fourteenth International Conference on Learning Representations (ICLR), (2026).

WEAR

https://mariusbock.github.io/wear/

Research has shown the complementarity of camera- and inertial-based data for modeling human activities, yet datasets with both egocentric video and inertial-based sensor data remain scarce. In this paper, we introduce WEAR, an outdoor sports dataset for both vision- and inertial-based human activity recognition (HAR). Data from 22 participants performing a total of 18 different workout activities was collected with synchronized inertial (acceleration) and camera (egocentric video) data recorded at 11 different outside locations. WEAR provides a challenging prediction scenario in changing outdoor environments using a sensor placement, in line with recent trends in real-world applications. Benchmark results show that through our sensor placement, each modality interestingly offers complementary strengths and weaknesses in their prediction performance. Further, in light of the recent success of single-stage Temporal Action Localization (TAL) models, we demonstrate their versatility of not only being trained using visual data, but also using raw inertial data and being capable to fuse both modalities by means of simple concatenation.

Further reading:
Marius Bock, Hilde Kuehne, Kristof Van Laerhoven, Michael Moeller: WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition. In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 8, Issue 4, Article No.: 175, pp. 1-21 (2024).

OpticsBench

https://github.com/PatMue/classification_robustness
https://github.com/PatMue/opticsbench_generate

Deep neural networks (DNNs) are increasingly deployed in safety-critical computer vision tasks. Therefore, vision models have to behave robustly to disturbances such as noise or blur. While seminal benchmarks exist to evaluate model robustness to diverse corruptions, blur is often approximated in an overly simplistic way to model defocus, while ignoring the different blur kernel shapes that result from lenses. To bridge this gap, we introduce OpticsBench, which evaluates primary aberrations such as coma, defocus, and astigmatism, i.e. aberrations that can be represented by varying a single parameter of Zernike polynomials. In addition, the provided code allows to create user-defined aberrations to investigate e.g. the model robustness to a specific lens. OpticsAugment effectively enhances model robustness to optical blur and common corruptions. 

Further reading:
Patrick Müller, Alexander Braun, Margret Keuper: Examining the Impact of Optical Aberrations to Image Classification and Object Detection Models. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) Vol. 48, Issue 3, pp. 2139-2153 (2026).
Patrick Müller, Alexander Braun, Margret Keuper: Classification Robustness to Common Optical Aberrations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 3632-3643 (2023).

AWESOME

https://github.com/jp-schneider/awesome

Image segmentation has greatly advanced over the past ten years. Yet, even the most recent techniques face difficulties producing good results in challenging situations, e.g., if training data are scarce, out-of-distribution examples need to be segmented, or if objects are occluded. In such situations, the inclusion of (geometric) constraints can improve the segmentation quality significantly. In this paper, we study the constraint of the segmented region being segmented convex. Unlike prior work that encourages such a property with computationally expensive penalties on segmentation masks represented explicitly on a grid of pixels, our work is the first to consider an implicit representation. Specifically, we represent the segmentation as a parameterized function that maps spatial coordinates to the likeliness of a pixel belonging to the fore- or background. This enables us to provably ensure the convexity of the segmented regions with the help of input convex neural networks. Numerical experiments demonstrate how to encourage explicit and implicit representations to match in order to benefit from the convexity constraints in several challenging segmentation scenarios.

Further Reading:
Jan Philipp Schneider, Mishal Fatima, Jovita Lukasik, Andreas Kolb, Margret Keuper, Michael Moeller: Implicit representations for constrained image segmentation. In International Conference on Machine Learning (ICML), pp. 43765-43790 (2024).
Jan Philipp Schneider, Mishal Fatima, Jovita Lukasik, Andreas Kolb, Margret Keuper, Michael Moeller: Implicit Representations for Image Segmentation. In UniReps Workshop: Unifying Representations in Neural Models. (2023).

DEFECT-GFRT

https://doi.org/10.5281/zenodo.18863663

This dataset contains experimental FMCW terahertz imaging measurements of glass fiber reinforced thermoplastic (GFRT) composites for non-destructive testing. It includes raw THz data and processing scripts for analyzing samples with and without defects such as delamination and consolidation variations. The dataset was first introduced in the following publication:

Further Reading:
A. Souliman, M. Kahl, D. Stock, M. Möller, B. Engel and P. H. Bolívar: Defect Detection in Bidirectional Glass Fabric Reinforced Thermoplastics Based on 3-D-THz Imaging. in IEEE Transactions on Terahertz Science and Technology, vol. 13, no. 3, pp. 209-220, May 2023, doi: 10.1109/TTHZ.2023.3247609.

CWFS

https://github.com/Muhammad-Kazim/Deep-Coded-Wavefront-Sensing—Bridging-the-Simulation-Experiment-Gap

Coded wavefront sensing (CWFS) is a recent computational quantitative phase imaging technique that enables one-shot phase retrieval of biological and other phase specimens. CWFS is readily integrable with standard laboratory microscopes and does not require specialized labor for its usage. The CWFS phase retrieval method is inspired by optical flow, but uses conventional optimization techniques. A main reason for this is the lack of publicly available datasets for CWFS, which prevents researchers from using deep neural networks in CWFS. In this paper, we present a forward model that utilizes wave optics to generate SynthBeads: a CWFS dataset obtained by modeling the complete experimental setup, including wave propagation through refractive index (RI) volumes of spherical microbeads, a standard microscope, and the phase mask, which is a key component of CWFS, with high fidelity. We show that our forward model enables deep CWFS, where pre-trained optical flow networks finetuned on SynthBeads successfully generalize to our SynthCell dataset, experimental microbead measurements, and, remarkably, complex biological specimens, providing quantitative phase estimates and thereby bridging the simulation-experiment gap.

Further Reading:
S. M. Kazim, P. Müller, and I. Ihrke: Deep Coded Wavefront Sensing: Bridging the Simulation–Experiment Gap. NeurIPS 2025 Workshop: Learning to Sense.

DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions

CosPGD: an efficient white-box adversarial attack for pixel-wise prediction tasks

https://github.com/shashankskagnihotri/cospgd

Adversarial robustness research has largely focused on image classification, while pixel-wise prediction tasks such as semantic segmentation, optical flow, and disparity estimation remain comparatively underexplored. To support research in this direction, we introduce CosPGD, an efficient white-box adversarial attack for dense prediction tasks. By providing a practical and effective attack formulation tailored to pixel-wise outputs, CosPGD enables systematic robustness evaluation of models whose predictions are structured over the full image domain rather than a single label.

Further Reading: 
Shashank Agnihotri, Steffen Jung, Margret Keuper: CosPGD: an efficient white-box adversarial attack for pixel-wise prediction tasks. Proceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024.

SemSegBench & DetecBench: Benchmarking Reliability and Generalization Beyond Classification

https://github.com/shashankskagnihotri/benchmarking_reliability_generalization

FlowBench: Benchmarking Optical Flow Estimation Methods for Reliability and Generalization

A Granular Study of Safety Pretraining under Model Abliteration

https://github.com/shashankskagnihotri/safety_pretraining

Safety alignment in open-weight language models is often evaluated only at the final model stage, making it difficult to understand which parts of safety pretraining remain effective under post-hoc editing interventions. To support research in this direction, we present a granular study of safety pretraining under model abliteration. By analyzing a sequence of safety-pretrained checkpoints, the work enables systematic study of how different safety properties evolve and which of them remain robust when refusal-related behaviors are removed through lightweight model edits.

Further reading: 
Shashank Agnihotri*, Jonas Jakubassa*, Priyam Dey, Sachin Goyal, Bernt Schiele, R. Venkatesh Babu, Margret Keuper: A Granular Study of Safety Pretraining
under Model Abliteration. 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop.