masud

Md Masudur Rahman


I am interested in Artificial Intelligence, Reinforcement Learning, and Robotics. My work focuses on designing and developing intelligent learning agents capable of making interpretable, critical decisions under uncertain conditions. Currently, my efforts are concentrated on the issue of generalization in reinforcement learning. I aim to develop algorithms that remain robust against the effects of confounders, with the ultimate goal of using these algorithms to solve real-world tasks.

I am a Ph.D. Candidate in the department of Computer Science at the Purdue University. My advisor is Professor Yexiang Xue.

I completed my M.S. from the department of Computer Science at the University of Virginia in 2018. Before joining the University of Virginia, I worked as a Lecturer in the Computer Science and Engineering department at the BRAC University. I completed my B.Sc. in Computer Science and Engineering from Bangladesh University of Engineering and Technology (BUET) in 2013.

Email: rahman64@purdue.edu
[Home] [Research] [Publications] [Teaching] [Mentoring] [Service] [CV] [Google Scholar]

Research

My research stands at the intersection of theoretical RL and practical applications, striving to bridge the gap between the two. The challenges in real-world applications of RL include dealing with imprecise state representations and the environment's shifts in data distribution. My objective is to leverage RL theory's strengths to address these challenges, creating algorithms that are grounded in realistic assumptions and are adaptable to shifting data distributions.

Research Topic Image Research Topic Image

Reinforcement Learning in the Presence of Confounders

The project develops advanced policy optimization algorithms to address the challenges of deploying reinforcement learning agents in complex real-world environments, focusing on improving their generalization to new, unseen situations.

  • [NAACL 2024][FMDM@NeurIPS Workshop 2023] Natural Language-based State Representation in Deep Reinforcement Learning. [pdf]
  • [ECML-PKDD 2022] Bootstrap State Representation using Style Transfer for Better Generalization in Deep Reinforcement Learning. [arXiv] [pdf][Code][video]
  • [ICMLA 2022] Bootstrap Advantage Estimation for Policy Optimization in Reinforcement Learning.[arXiv] [pdf][Code] [video]
  • [RRL@ICLR Workshop 2023] Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learning. [pdf]
  • [Preprint 2023] Adversarial Style Transfer for Robust Policy Optimization in Deep Reinforcement Learning. [pdf]
  • [Preprint 2023] Adversarial Policy Optimization in Deep Reinforcement Learning. [pdf]


    • Research Topic Image Research Topic Image

    Robust Policy Optimization for Efficient Deep Reinforcement Learning

    The project aims to enhance deep reinforcement learning by diversifying training datasets and enriching data trajectories. This approach is designed to facilitate the development of robust policies and improve adaptability to new situations with fewer data requirements, enabling effective application across diverse and dynamic environments.

  • [Preprint 2023] Robust Policy Optimization in Deep Reinforcement Learning. [pdf] [Experiments]
  • [Implementation] CleanRL Library Integration. [Documentation] [Code] [Twitter Announcement]
  • [Implementation] sklr Library Integration. [Documentation] [Code (PyTorch)] [Code (Jax)]


    • Research Topic Image Research Topic Image

    Advancing Robotic Surgery with Semi-autonomous Teleoperation

    The project aims to advance the application of machine learning in safety-critical systems, focusing on developing a semi-autonomous teleoperated robotic surgery system to overcome the limitations of data scarcity.

  • [ICRA 2021] DESERTS: Delay-Tolerant Semi-Autonomous Robot Teleoperation for Surgery. [pdf]
  • [IROS 2019] DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots. [pdf] [arXiv]
  • [RO-MAN 2021] Sequential Prediction with Logic Constraints for Surgical Robotic Activity Recognition. [pdf] [code] [video]
  • [CMBBE Journal 2020] SARTRES: A Semi-Autonomous Robot TeleopeRation Environment for Surgery. [pdf]
  • [Military Medicine Journal 2020] From the DESK (Dexterous Surgical Skill) to the Battlefield - A Robotics Exploratory Study [Link]
  • [RO-MAN 2021] Dexterous Skill Transfer between Surgical Procedures for Teleoperated Robotic Surgery. [pdf]
  • [RO-MAN 2019] Transferring Dexterous Surgical Skill Knowledge between Robots for Semi-autonomous Teleoperation. [pdf]

    • Research Topic Image

    Automated Burn Diagnostic System for Healthcare (AMBUSH)

  • Approximately 1.25 million people receive treatment for burns each year, with 40,000 being hospitalized for these injuries in the United States, resulting in medical costs of approximately $7.9 billion annually.
  • This project aims to enhance the accuracy and efficacy of burn depth assessment, a critical yet challenging aspect of burn injury treatment. The ultimate goal is to utilize multimodal data, including digital photographs and ultrasound (TDI, B-Mode) data, to develop an AI-based burn assessment system. This system will help predict burn depth and provide explanations to expert surgeons, enabling them to make better treatment decisions for critical patients.
  • Technically, this approach leverages a large-scale foundational model (e.g., Vision Transformer) for burn depth prediction. It utilizes vision-language (VLM) and large-language models (LLM) to generate explanations for doctors, assisting in treatment for improved accuracy. This will facilitate early burn diagnosis and intervention.
  • I am the student team lead of this project, which is an interdisciplinary collaboration with the Department of Computer Science and School of Industrial Engineering at Purdue University and the Department of Surgery, School of Medicine, University of Pittsburgh.
  • [Media Coverage: IE@Purdue News ]

    • Publications

    [C15] Natural Language-based State Representation in Deep Reinforcement Learning. [pdf]

    NAACL 2024, Annual Conference of the North American Chapter of the Association for Computational Linguistics. conference

    [Preprint 2024] Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning. [arXiv][pdf] [GitHub][Website]


    [W4] Natural Language-based State Representation in Deep Reinforcement Learning. [pdf]

    FMDM@NeurIPS 2023, Foundation Models for Decision Making Workshop at the thirty-seventh Annual Conference on Neural Information Processing Systems. workshop

    [W3] Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learning. [pdf]

    RRL@ICLR 2023, Workshop on Reincarnating Reinforcement Learning at he Eleventh International Conference on Learning Representations. workshop

    [C14] Bootstrap Advantage Estimation for Policy Optimization in Reinforcement Learning. [arXiv] [pdf] [video]
    ICMLA 2022, In Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA), 2022. conference

    [C13] Bootstrap State Representation using Style Transfer for Better Generalization in Deep Reinforcement Learning. [arXiv] [pdf][video]
    ECML-PKDD 2022, In Proceedings of the 2022 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2022. conference

    [C12] ASAP: A Semi-Autonomous Precise robotic framework for remote surgery under delays.
    MHSRS 2022 , In Military Health System Research Symposium, 2022. Abstract Paper (Oral Presentation)

    [C11] Sequential Prediction with Logic Constraints for Surgical Robotic Activity Recognition. [pdf] [code] [video]
    RO-MAN 2021, In Proceedings of the 30th IEEE International Conference on Robot & Human Interactive Communication - 2021, 8 pages. conference

    [C10] Dexterous Skill Transfer between Surgical Procedures for Teleoperated Robotic Surgery. [pdf]
    RO-MAN 2021, In Proceedings of the 30th IEEE International Conference on Robot & Human Interactive Communication - 2021, 7 pages. conference

    [C9] A Semi-autonomous Robotic Framework for Remote Surgery under Delays.
    MHSRS 2021 , In Military Health System Research Symposium, 2021. Abstract Paper (Oral Presentation)

    [C8] DESERTS: Delay-Tolerant Semi-Autonomous Robot Teleoperation for Surgery. [pdf]
    ICRA 2021, In Proceedings of the 2021 IEEE International Conference on Robotics and Automation - 2021, 8 pages. conference

    [J3] SARTRES: A Semi-Autonomous Robot TeleopeRation Environment for Surgery. [pdf]
    AECAI, TCIV 2020, Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization Journal - 2020, AE-CAI | CARE | OR 2.0 Joint MICCAI Workshop - 2020, 14 pages. journal

    [J2] From the DESK (Dexterous Surgical Skill) to the Battlefield - A Robotics Exploratory Study
    MHSRS Journal (Military Medicine) 2020, In Military Health System Research Symposium (Military Medicine), 2020 23 pages. journal

    [C7] ASTRO: A Semi-Autonomous Telemedicine Robot for Operative Surgery.
    MHSRS 2020, In Military Health System Research Symposium, 2020. 3 pages. Abstract Paper

    [C6] Transferring Dexterous Surgical Skill Knowledge between Robots for Semi-autonomous Teleoperation. [pdf]
    Ro-Man 2019, In Proceedings of the 28th IEEE International Conference on Robot and Human Interactive Communication, 2019, 6 pages. conference

    [W2] Morality in Decision-Making: A Causal Approach. [video]

    MoDeM 2019, RLDM Workshop on Moral Decision Making [Link] , 2019. talk workshop

    [C5] DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots. [pdf] [arXiv]

    IROS 2019, In Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, 8 pages. conference

    [C4] Toward Optimal Selection of Information Retrieval Models for Software Engineering Tasks. [pdf]
    SCAM 2019, In Proceedings of the 19th IEEE International Working Conference on Source Code Analysis and Manipulation, 2019, 12 pages. conference

    [J1] Recommending GitHub Projects for Developer Onboarding. [pdf] [link]

    IEEE Access 2018, 13 pages. journal

    [C3] Evaluating How Developers Use General-Purpose Web-Search for Code Retrieval. [pdf] [arXiv] [slides] [code]

    MSR 2018, In Proceedings of the 15th International Conference on Mining Software Repositories, 11 pages. conference

    [C2] Which Similarity Metric to Use for Software Documents? A study on Information Retrieval based Software Engineering Tasks. [pdf] [poster]

    ICSE 2018 Companion, In Proceedings of 40th International Conference on Software Engineering Companion, 2018, 2 pages. conference

    [W1] Finding Similar Projects in GitHub using Word2Vec and WMD. [slides]

    NL+SE @FSE 2016 talk workshop

    [C1] Topic Model based Privacy Protection in Personalized Web Search. [pdf] [code]

    SIGIR 2016, In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2016, 4 pages. conference

    [T1] A Case Study on the Impact of Similarity Measure on Information Retrieval based Software Engineering Tasks. [pdf][arXiv]

    Technical Report 2018, 22 pages. technical report

    Teaching

    Teaching Assistant Instructor

    Mentoring

    I am fortunate to work with following students in various capacities.

    Service

    Conference Reviewer Journal Reviewer Sub-reviewer
    Email: rahman64@purdue.edu
    [Home] [Research] [Publications] [Teaching] [Mentoring] [Service] [CV] [Google Scholar]