Hierarchies of reward machines

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Furelos Blanco, Daniel
• dc.contributor.author Law, Mark
• dc.contributor.author Jonsson, Anders
• dc.contributor.author Broda, Krysia
• dc.contributor.author Russo, Alessandra
• dc.date.accessioned 2025-01-27T13:54:20Z
• dc.date.available 2025-01-27T13:54:20Z
• dc.date.issued 2023
• dc.description.abstract Reward machines (RMs) are a recent formalism for representing the reward function of a reinforcement learning task through a finite-state machine whose edges encode subgoals of the task using high-level events. The structure of RMs enables the decomposition of a task into simpler and independently solvable subtasks that help tackle longhorizon and/or sparse reward tasks. We propose a formalism for further abstracting the subtask structure by endowing an RM with the ability to call other RMs, thus composing a hierarchy of RMs (HRM). We exploit HRMs by treating each call to an RM as an independently solvable subtask using the options framework, and describe a curriculum-based method to learn HRMs from traces observed by the agent. Our experiments reveal that exploiting a handcrafted HRM leads to faster convergence than with a flat HRM, and that learning an HRM is feasible in cases where its equivalent flat representation is not.
• dc.description.sponsorship Anders Jonsson is partially funded by TAILOR, AGAUR SGR and Spanish grant PID2019-108141GB-I00
• dc.format.mimetype application/pdf
• dc.identifier.citation Furelos-Blanco D, Law M, Jonsson A, Broda K, Russo A. Hierarchies of reward machines. In: Krause A, Brunskill E, Cho K, Engelhardt B, Sabato S, Scarlett J, editors. Proceedings of the 40th International Conference on Machine Learning, PMLR; 2023 Jul 23-29; Honolulu, Hawaii, USA. San Diego; 2023. p.10494-541
• dc.identifier.doi https://doi.org/10.48550/arXiv.2205.15752
• dc.identifier.uri http://hdl.handle.net/10230/69309
• dc.language.iso eng
• dc.publisher PMLR
• dc.relation.projectID info:eu-repo/grantAgreement/ES/2PE/PID2019-108141GB-I00
• dc.rights Copyright 2023 by the author(s).
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.subject.keyword Reward machines
• dc.subject.keyword Hierarchies
• dc.title Hierarchies of reward machines
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion