e-Repositori UPF :: Erasmus Mundus joint Master in Artificial Intelligence (EMAI). Master thesis projects

Recent Submissions

Face de-identification using diffusion models
(2025) Gonzalez Hernandez, Manfred
In an era of increasing surveillance and data sharing, protecting individuals’ identities in visual data has become a critical concern. This thesis addresses the problem of face de-identification using diffusion models, a powerful class of generative models capable of synthesizing high-fidelity images. We propose two novel approaches that modify the denoising trajectory of diffusion models through two different identity-independent and identity dependent guidance regimes to generate de-identified facial images while preserving essential attributes such as expression and gender. Our method lever ages pre-trained identity recognition models and preconditioned guidance to balance visual realism/quality and privacy. We evaluate our approach on four benchmark datasets—RaFD, XM2VTS, LFW, and CALFW—using metrics such as AUC, F1 Score, Fr ́echet Distance, and Mean Squared Error. The results demonstrate comparable performance with state-of-the-art techniques in both identity removal effectiveness and attribute preservation. Additionally, we analyze the behavior of the diffusion process under different guidance strategies, providing insight into the trade-offs between identity concealment and image quality.
A Novel method for open-vocabulary panoptic segmentation
(2025) Kormushev, Nikolay
Open-vocabulary panoptic segmentation aims to segment and classify visual content into both known and unseen categories using natural language supervision. While class-agnostic mask generators produce reasonably high quality masks, this thesis identifies two main bottlenecks limiting performance: mask quality assessment, where valid masks are often mistakenly discarded as background, and semantic classification, which remains challenging especially for unseen categories. To address these, we propose a two part solution: a novel background mask reclassification module that recovers valid masks misclassified as background, and a CLIP fine-tuning strategy that preserves alignment between visual and textual embeddings. Together, these methods improve panoptic quality (PQ) on ADE20K from a baseline of 26.6 to 27.94, with analysis showing that addressing errors in mask quality and semantic classification could theoretically increase PQ to 65.9. These findings offer practical advancements and valuable insights toward bridging the gap between open- and closed-vocabulary segmentation.
Deep learning for predicting wave parameters from wind components in the Adriatic basin
(2025) Lendering, Camile Ruben
Predicting high-resolution ocean wave parameters, such as significant wave height, mean wave period, and direction, in complex coastal regions like the Adriatic Sea is essential but computationally intensive when using traditional physical models. This thesis explores deep learning-based statistical downscaling, using coarse-resolution ERA5 wind fields to generate fine-scale wave predictions. Both deterministic (U-Net, ClimaX) and probabilistic (WGAN-GP, Conditional Flow Matching) models were developed and compared. Results show that deep learning can effectively model the nonlinear relationships between wind and waves. Among the tested approaches, the CFM model achieved the highest accuracy for ensemble mean predictions and offered reliable uncertainty quantification, highlighting its potential for efficient and scalable high-resolution wave forecasting.
Deep learning for precipitation nowcasting in Slovenia
(2025) Bulcao Ribeiro, Bernardo Perrone De Menezes
Accurate probabilistic precipitation nowcasting remains a major challenge due to the inherent uncertainty and complexity of atmospheric systems, which deterministic models often fail to capture. This thesis addresses this gap by introducing Conditional Flow Matching (CFM), a novel generative modeling approach, to the task of probabilistic nowcasting. We adapt state-of-the-art deep learning architectures to serve as the backbone for CFM, enabling the generation of diverse, high-fidelity ensemble forecasts. Our method achieves state-of-the-art performance on the SEVIR dataset, with a 15% improvement in CRPS over strong baselines like CasCast. We further validate the approach on the ARSO dataset, curated for nowcasting in Slovenia, where transfer learning from SEVIR yields consistent performance gains. Both qualitative and quantitative results demonstrate that CFM produces sharp, reliable, and spatially coherent forecasts, thus advancing the state of probabilistic nowcasting.
ML-based attack on module-LWE assessing the security of module-lattice-based schemes
(2025) Bassotto, Cristian
The dawn of quantum computing compromises the security foundations of classical public-key cryptography, motivating NIST’s recent standardization of post-quantum cryptographic schemes. Two of the four selected algorithms are based on the hardness of the Module Learning With Errors (Module-LWE) problem. As Module-LWE becomes the mathematical primitive of next-generation cryptographic standards, its robustness needs to be evaluated not just against classical and quantum algorithms, but also against novel AI-powered attacks. In this work, we introduce NoMod ML-Attack, a hybrid white-box cryptanalytic attack that avoids the difficulty of directly modelling modular reduction in Module-LWE. Instead, NoMod views modular wraparounds as a type of statistical corruption and reformulates secret recovery as a robust linear estimation problem. Our method begins with advanced lattice preprocessing, improved using several optimizations. In particular, we propose (i) a reduced-vector saving strategy that accumulates and reuses short vectors during tours, and (ii) an algebraic amplification technique that exploits Ring-LWE automorphisms to expand the pool of usable samples. After this preprocessing step, we train robust linear estimators based on Tukey’s Biweight loss, prioritizing direct secret recovery and sampleefficient methods over transformer-based architectures. Our experimental tests demonstrate that NoMod broadens the range of ML-based cryptanalysis. We achieve complete recovery of binary secrets for dimension n = 350, recovery of sparse binomial secrets at n = 256, and successful recovery of sparse secrets in CRYSTALS-Kyber settings with parameters (n,k) = (128,3) and (256,2). Throughout these regimes, NoMod outperforms classical lattice-only techniques and, in some instances, produces results competitive with transformer-based frameworks, such as SALSA [1] and SALSA PICANTE [2]. Finally, to enable future research, we release our open-source implementation of NoMod ML-Attack to support continued study and benchmarking.
Towards trustworthy AI research assistants: leveraging knowledge graphs for knowledge synthesis
(2025) Çalış, Ahmet
The use of LLMs is becoming more widespread every day. Its use of purpose varies from answering questions, evaluating information, and summarizing a large amount of information. This capability of LLMs also helps researchers who need to read and follow a lot of papers. Making sense of scientific research is more important than ever for informed decisionmaking. It is becoming nearly impossible for researchers to keep up and piece everything together manually with the overwhelming number of new studies being published every day. AI-powered research assistants, especially those built on large language models are beginning to help to find, analyze, and summarize huge volumes of information. However, how can we be sure that these summaries are accurate and reliable? Recent studies point to a promising solution, the grounding of LLMs with structured formats such as knowledge graphs (KGs), which can help improve both the reliability, explainability, and quality of the information they produce. This thesis explores how to generate knowledge graphs from unstructured text, with a focus on understanding and comparing different methods and configurations. The main goal is to evaluate how well these approaches capture useful information, both in terms of quantity and quality, by using a recently published benchmark as a point of reference. Ultimately, the findings are designed to support the integration of knowledge graphs into LLM-based research assistants, helping to make knowledge synthesis more accurate, reliable, and effective.
From perception to action: implementing in-context imitation learning on a franka robot for pick-and-place tasks
(2025) Carpes Martínez, Antonio Alberto
This thesis presents a practical implementation of Instant Policy, an In-Context Imitation Learning (ICIL) model characterized by the rapid learning of new tasks, after processing a few number of demonstrations at inference time. The research evaluates how demonstration context modifications affect the model ability to understand and generalize manipulation behaviors using a Franka Emika Panda arm and Intel RealSense D435 camera integrated with Instant Policy, a state-of-the-art one-shot learning model. The core research systematically modifies demonstration buffers to analyze the model contextual reasoning capabilities across different pick-and-place scenarios. Besides, we deploy a modular pipeline that transforms RGB-D input into structured point clouds through YOLOv11-based segmentation, enabling object identification, demonstration extraction and model deployment at test time. To address gripper annotation challenges, we introduce an automated dataset creation methodology combining LangSAM for text-prompt-based segmentation and XMem++ for video mask propagation. The control architecture employs Instant Policy as a Denoising Diffusion Implicit Model, generating action sequences through graph-based reasoning over point clouds and demonstration context. Experimental results demonstrate successful adaptation of pick-and-place behaviors based on different demonstration contexts, with generalization across object pose and background variations. Performance analysis reveals critical dependencies on segmentation quality, highlighting robust perception requirements for real-world deployment. This work validates ICIL viability for robotic pick-and-place tasks, contributing insights into context understanding, automated dataset creation, and empirical validation of ICIL performance in unstructured manipulation scenarios.
Field identification in low-resolution satellite images: a performance analysis of semantic and instance segmentation models on the fields of the world dataset
(2025) Chissich, Roberta
This study investigates the challenges and performance of deep learning architectures for agricultural field segmentation. We used a large, globally distributed dataset with a vast variability in field to train and evaluate semantic and instance segmentation models. Specifically ResUNet for semantic segmentation and Mask R-CNN for instance segmentation. We then compared their results to baseline Unet models. ResUNet achieves strong results in the 2-class configuration, with 92% (macro) pixel accuracy and a (macro) IoU of 0.92, demonstrating robust fields identification. However, introducing a third, “boundary” class reduces (macro) IoU to 0.62, revealing that multi-class segmentation remains challenging in agricultural contexts. Mask R-CNN’s performance is more limited, achieving only an AP of 0.121 overall and 0.252 at IoU = 0.50. Part of this apparent underperformance comes from imperfect ground truth data, i.e. many “false positives” correspond to actual fields that are absent from the annotations. Nonetheless, the difficulty of handling irregular shapes and define fine boundaries remain major bottlenecks. Overall, ResUNet significantly improve the performance over the baseline Unet in both accuracy and speed. However, challenges related to class imbalance, imperfect annotations, boundary complexity, and resource constraints remain active areas for improvement. These findings highlight the need for better dataset curation in order to achieve better performances to segment agricultural landscapes.
Multimodal hierarchical masked autoencoders tailored to ECG time series for atrial fibrillation risk stratification
(2025) Cianci , Giuseppe
Background: Atrial fibrillation affects 26% of Europeans over 40, with catheter ablation being a primary treatment. Low voltage areas significantly influence ablation success, but their identification currently requires invasive electroanatomical mapping during the procedure. Objective: We developed a deep learning framework to predict finegrained, region-specific low voltage areas from electrocardiograms and clinical data, enabling pre-procedural risk stratification. Methods: We collected data from 138 atrial fibrillation patients undergoing catheter ablation at University Hospital Basel, with external validation cohorts from Lausanne (n=20) and Bern (n=9). Our multimodal architecture processes ECG signals through pretrained backbones (ST-MEM, CRNN, DenseNet) and clinical features through separate branches, fusing them for hierarchical multilabel classification across 8 atrial regions, 2 area thresholds, and 4 voltage thresholds (64 targets total). Key innovations include: (1) adaptation of max constraint loss to logit space for numerical stability, (2) parallel multi-lead QRS detection for robust heartbeat segmentation, and (3) region-specific adaptive learning rate scheduling. Results: Our model achieved an AUROC of 0.782 for general low voltage area prediction, outperforming models trained on 10 times more data from previous studies. A single model is enough to predict 64 classes of low voltage areas, which is a significant improvement over previous studies. Conclusions: Non-invasive low voltage area prediction from routine ECGs is feasible and clinically relevant. The hierarchical multilabel framework provides detailed spatial information about atrial substrate, potentially enabling better patient selection and procedural planning for atrial fibrillation ablation.
Steering vector-guided molecular generation using language models
(2025) Dimitrievikj, Aleksandar
Recent advances in chemical language models have enabled rapid exploration of chemical space through generative design of novel molecules. However, precise control over key molecular properties—such as size, aqueous solubility, and lipophilicity—remains challenging without retraining or introducing complex optimization steps. This thesis investigates a lightweight approach based on contrastive activation addition, where differences in model activations between molecules with favorable and unfavorable properties are used to compute steering vectors. These vectors are applied during generation to bias the model towards producing molecules with desired characteristics, without modifying model weights. Using a GPT-style molecular generator conditioned on protein targets, we demonstrate that steering can consistently shift molecular property distributions: reducing median heavy-atom counts, improving predicted solubility by up to 1.4 logS units, and increasing the fraction of molecules within the optimal lipophilicity window for oral drugs. The approach preserves high validity rates, typically above 90%, and requires minimal computation, making it suitable for early-stage drug discovery workflows. Two variants of the method are compared: a global steering vector applied uniformly, and a tokenaligned vector field adapting dynamically to each generation step. While the latter amplifies property shifts, it also increases the risk of generating invalid molecules under certain settings. Overall, this work demonstrates that activation steering offers an interpretable, low-overhead mechanism for fine-tuning molecular properties, providing a practical tool to accelerate the design–make–test cycle in drug development. Future directions include extending this strategy to multi-property optimization and models that capture three-dimensional molecular structures.
Federated fine-tuning of foundation models for human activity recognition using wearable data under EU regulations
(2025) Dorin , Dariana
Wearable health devices generate vast amounts of sensitive physiological data, creating opportunities for AI-driven health monitoring while raising significant privacy concerns under emerging European regulations. This thesis investigates the feasibility of federated learning for human activity recognition (HAR) using foundation models, specifically examining whether privacy-preserving approaches can maintain clinical accuracy while complying with GDPR, the EU AI Act, and the forthcoming European Health Data Space (EHDS). Three fine-tuning strategies are systematically compared for adapting the HARNet10 foundation model to the PAMAP2 dataset under both centralized and federated learning paradigms. Experiments reveal that the All-Layers-Except-BatchNorm (All-BN) strategy achieves optimal centralized performance, while classifier-only training provides a compelling balance of performance and stability. In federated settings implemented via PySyft, classifier-only training achieves just 3.33 percentage points below its centralized counterpart while maintaining identical effort-level classification accuracy (87.28%). The performance gap, though modest, is offset by the practical advantages of classifier-only adaptation, which offers minimal communication overhead, strong privacy preservation through local feature extraction, and compatibility with resource-constrained wearable devices. This work contributes with a reproducible benchmark for adapting foundation models under regulatory constraints and demonstrates that privacy-preserving federated training can deliver functional accuracy suitable for real-world health applications.
Evaluating feature matching and ensemble strategies for monocular pose estimation in colonoscopy videos
(2025) Duthie, Honor
Colonoscopy, a key procedure for colorectal cancer screening, could benefit from 3D reconstruction and pose estimation for enhanced navigation, but robust feature matching remains an open challenge due to tissue deformation, variable illumination, and motion artefacts. This thesis evaluates three state-of-the-art learned feature matchers (DISK-LightGlue, GIM-LightGlue, and XFeat) and an ensemble approach for monocular pose recovery in synthetic colonoscopy videos. Results show that while the ensemble achieved the lowest rotational error (0.56°) and failure rate (0.5%) on registered sequences, trajectory recovery remained poor, and screening video evaluation was inconclusive due to pipeline limitations. These f indings suggest that current matchers alone are insufficient for reliable reconstruction in this domain, highlighting the need for deformation-aware models and more representative data before clinical application is feasible. Code is available at https://github.com/hduthie/thesis-colonoscopy-eval
Self-supervised and in-context learning techniques for automated optical inspection
(2025) Figueira, Joaquín
Automated Optical Inspection (AOI) is a family of techniques used to find defects and anomalies in electronic devices from high-quality photographs of different regions of an integrated component and its packaging. Current methods use computer vision models and image preprocessing pipelines specific to each chip design and manufacturer. As a result, the current deep learning approach for AOI requires a long retraining process whenever new devices are introduced or significant covariate shifts occur in the input image distribution. In this work, we adapt and evaluate different pre-training techniques (DINO, iBOT, and MAE) for small vision transformers (ViT and FasterViT) to streamline the design process of AOI semantic segmentation models and shorten the training time needed to adapt the models to new input conditions. We use a custom, relatively small dataset for model pre-training with only 7000 unlabeled images, showing how the pre-training strategies perform well in small data regimes. Furthermore, we introduce a set of retrieval-based scene understanding techniques to solve the task of semantic segmentation of wire-bonded devices with virtually no training time in labeled data. Our results demonstrate how our custom pre-trained encoders and retrieval strategies outperform comparable convolutional architectures pre-trained using full supervision in semantic segmentation, both in speed and quality, when training time is constrained. Moreover, we show how our proposed image retrieval strategies generalize to existing ViT models pretrained on different datasets, and how the techniques can be used to predict images of a single device and produce high-quality segmentation masks using a relatively small number of labeled training images. Finally, we show how the retrieval strategies outperform fine-tuned, convolutional encoder-decoder models in the context of out-of-distribution, unseen images.
Attention-enhanced 3D convolutional neural network for automated lung nodule malignancy assessment
(2025) Lumburovska, Leonida
Accurate classification of lung nodule malignancy remains a critical challenge in medical imaging, directly impacting early lung cancer diagnosis and patient outcomes. While existing 3D convolutional neural networks have shown promise in capturing volumetric morphological characteristics, effectively focusing on the most discriminative features for malignancy classification remains challenging. This study investigates a customized 3D ResNet architecture enhanced with Convolutional Block Attention Module (CBAM) for improved lung nodule malignancy classification, with validation across multiple datasets to demonstrate generalizability. Our proposed framework integrates CBAM attention mechanisms into a 3D ResNet backbone, enabling the model to adaptively focus on both spatial and channel-wise features critical for malignancy determination. The CBAM module allows the network to emphasize informative features while suppressing irrelevant noise commonly present in CT imaging. This attention-guided approach addresses the inherent challenge of extracting discriminative features from volumetric medical data where subtle morphological differences often determine malignancy status. The 3DResNetarchitecture uses residual connections to facilitate deep feature learning while reducing gradient vanishing problems, particularly important when processing high-dimensional volumetric data. By operating directly on 3D patches, our model captures spatial relationships across all three dimensions, providing a more comprehensive representation of nodule characteristics including shape irregularities, internal texture patterns, and boundary definitions that are crucial malignancy indicators. To enhance model interpretability and validate attention mechanisms, we generate spatial attention maps that visualize regions of focus during classification decisions. These attention maps demonstrate that our CBAM-enhanced model successfully identifies clinically relevant anatomical structures and morphological features associated with malignant nodules. Additionally, we conducted texture feature experiments to complement the deep learning approach. Traditional texture analysis methods were employed to extracthandcrafted features such as gray-level co-occurrence matrix (GLCM) statistics, as well as classification of nodules based on texture. These experiments provide additional insights into nodule characteristics, and when combined with the spatial attention maps generated by our model, offer a complementary view of how both traditional handcrafted features and learned attention mechanisms contribute to understanding nodule texture patterns and properties. Experimental validation was performed across multiple lung nodule datasets, both public and private, demonstrating the generalizability of our approach. The crossdataset evaluation confirms that the architecture maintains robust performance across different imaging protocols, patient populations etc., as well as consistence in all metrics across all tested datasets. The spatial attention maps revealed that the model correctly learns to focus on the relevant regions, aligning with radiological expertise and improving clinical interpretability. This work contributes to the advancement of computer-aided diagnosis systems for lung cancer screening by demonstrating the effectiveness of attention-enhanced 3D CNNs. The proposed CBAM-integrated 3D ResNet architecture offers a promising solution for automated lung nodule malignancy classification, potentially supporting radiologists in making more accurate and timely diagnostic decisions.
Deep research agentic framework for mitigating bias in AI-driven healthcare diagnostics
(2025) Ferreira Moreira, Pedro José
Transformer-scale language models can now ace many medical exams, but their frozen parametric memory risks propagating outdated guidelines and systemic bias to the bedside. To counter this, we re-imagine the diagnostic assistant as a navigator that plans, retrieves, executes code, and verifies evidence rather than guessing from memory. We introduce DeepMed, a 4 B-parameter multi-agent framework who attempts to switch the paradigm of medical assistances from diagnostic oracles to information retrievers. Agents invoke external tools via the open Model Context Protocol (MCP), including M3, a natural-language gateway to the MIMIC-IV EHR, and a sandboxed Python REPL for on-the-fly calculations. Performance is audited on the newly proposed MedBrowseComp benchmark (1 089 quarterly-regenerating, multi-hop oncology related queries), legacy QA suites, the EquityMedQA counter factual set, and the EHRSQL challenge. With just a 4 billion parameter LLM as the cognitive engine DeepMed achieves 26 %single-pass accuracy on MedBrowseComp, outperforming larger entreprise grade systems that rely on 10 to 100 times larger fine tuned models while running locally on a consumer laptop. On EquityMedQA it increases correctness from 50.8 % to 57.4%, a 13% relative reduction in demographic disparity. Coupling MCP to the M3 EHR interface lifts pass@1 on EHRSQL from 2% to 9%. By fusing agentic planning, typed tool use, and evidence-first reporting, DeepMed shows that bias-aware, verifiable clinical AI can be achieved without frontier-scale models or costly GPU clusters. The open-sourced multiagent framework, MCP server tool contributions like M3 and MedBrowseComp benchmark provide a reproducible path toward transparent, low-cost decision support in safety-critical healthcare settings.
Multi-turn attacks for automated LLM red teaming
(2025) Niayesh, Nazanin
Large language models (LLMs) are increasingly used in applications within various domains such as healthcare, research, and education. With the growing use of these models, especially in critical systems (e.g., self-driving cars), the security of these models becomes increasingly crucial. Automated red teaming aims to efficiently and effectively uncover the security vulnerabilities of LLMs and LLM-based applications so that these can be mitigated before being misused by malicious parties. Red teaming often utilizes jailbreak attacks. Most works on jailbreak attacks in the literature focus on single-turn attacks, which are executed as a single input prompt to the target LLM within one conversation turn. Multi-turn attacks, however, better represent manual red teaming, which is also often done over several conversation turns. Additionally, multi-turn attacks have been shown to uncover a greater number and variety of weaknesses and security vulnerabilities compared to single-turn attacks. While research on multi-turn attacks has increased recently, most works on automatic jailbreaking are still centered on single-turn techniques. This work aims to contribute to the improvement and spreading of multi-turn attacks such that they can be used in automated red teaming to improve the security of LLM applications. The focus of this work lies on the automatic black-box multi-turn attack Generative Offensive Agent Tester (GOAT). Specifically, we extend Generative Offensive Agent Tester (GOAT) with new jailbreak strategies, and both GOAT and these additional strategies are implemented in the Azure Python Risk Identification Tool (PyRIT) red teaming and security evaluation framework. Extensive experiments are conducted to compare the performance of the proposed changes in relation to the original GOAT method in a variety of setups involving a number of tuned attackers and scorer models. A comprehensive analysis of these experimental results reveals that the proposed modifications result in promising improvements in most setups.
Overcoming operational barriers to scalable school detection using low-cost satellite imagery and cross-country knowledge distillation
(2025) Pattichis, Andreas
Over 2.2 billion children lack internet access at home. To address this critical gap, UNICEF and ITU’s Giga initiative aims to connect every school to the internet by 2030 using AI-powered school mapping that relies on satellite imagery. However, Giga’s existing weakly supervised deep learning workflow faces significant challenges when scaling to low- and middle-income countries. The pipeline uses expensive high-resolution Maxar satellite imagery. Moreover, preparing each country’s training dataset demands extensive manual validation, often requiring several days of expert time. To overcome these limitations, this research extends the established pipeline through three successive approaches for end-to-end improvement. First, we evaluate Mapbox as a lower-cost imagery alternative. Second, we build cross country ensembles that eliminate target-country training requirements by leveraging neighboring-country models. Third, we distill knowledge from these ensembles to reduce manual validation requirements by automatically generating soft labels for target-country student models. Our experimental evaluation across three SubSaharan African countries reveals three key findings. Mapbox imagery maintains performance within 10 % of Maxar baselines, providing a cost-effective alternative. Cross-country ensembles achieve comparable performance to target-country baselines while eliminating the need for target-country labels. Most significantly, Knowledge Distillation nearly matches target-country baseline performance, outperforms cross-country ensembles, and auto-generates soft labels for 82–88 % of school samples reducing manual validation from days to hours. These findings establish a practical framework that cuts the cost and expertise needed for AI-driven school mapping, marking substantial progress toward sustainable local ownership and Giga’s 2030 goal of universal school connectivity.
Set covering machine on t-cell receptor LLM representations for lung cancer prediction
(2025) Vegas Morales, Neus
T-cell receptors (TCRs) provide insights into immune recognition of cancer. We explore whether interpretable rule-based classifiers derived from SCEPTR embeddings of TCR sequences can differentiate cancer repertoires from healthy controls. Using the Set Covering Machine algorithm, we developed models with hyperplane and similarity based rules across alpha and beta chains. Despite strong performance on training data, models failed to generalize to external datasets. Unexpectedly, alpha-chain models often outperformed beta-chain models, and single rules sometimes achieved high training accuracy, suggesting overfitting. Our findings highlight challenges in detecting cancer-specific TCR signatures and indicate current embeddings may capture technical patterns rather than biological signal. We propose future directions including improved rule generation strategies and validation with functionally annotated repertoires.

Navigate

Browse

Recent Submissions