Paul Liang, MIT
Paul Pu Liang
Assistant Professor, MIT Media Lab and MIT EECS
Email: ppliang(at)mit.edu
[CV]
[google scholar]
[linkedin]
[github]
[twitter]
[instagram]
I will be joining MIT as an Assistant Professor in Fall 2024, joint between the MIT Media Lab and MIT EECS.
I will direct the new multisensory intelligence research group, which studies the foundations of multisensory AI and its impact on the human experience. My group will work on:
(1) Foundations of multisensory AI: The science and engineering of AI systems that can learn and interact with the world through integrating diverse sensory channels such as text, speech, audio, video, physical sensors, and physiological messages.
(2) Impact on the human experience: AI for human physical, emotional, and social well-being, multimedia generative AI to augment human creativity, climate and environment sensing, enhancing human sensory experiences like music, art, cultures, smell, and taste.
(3) Real-world human-AI interaction: Ensuring fairness, robustness, trust, privacy, and efficiency for responsible deployment.
I will be spending Summer 2024 as a visiting researcher in the AI, psychology, and neuroscience program at UC Berkeley's Simons Institute for the Theory of Computing.
I recently received my Ph.D. from the Machine Learning Department at Carnegie Mellon University, advised by Louis-Philippe Morency and Ruslan Salakhutdinov.
I was also fortunate to collaborate with Manuel Blum, Lenore Blum, Faisal Mahmood, Jack Hessel, and Yejin Choi at Berkeley, Harvard Medical School, and UW/AI2.
My research was generously supported by a Siebel Scholars Award, Waibel Presidential Fellowship, Facebook PhD Fellowship, and Center for Machine Learning and Health Fellowship,
and has been recognized by 4 best paper/honorable mention awards at international conferences and workshops.
I love teaching and was honored to receive the Alan J. Perlis Graduate Student Teaching Award for co-instructing courses on multimodal machine learning.
Previously, I received an M.S. in Machine Learning and a B.S. with University Honors in Computer Science and Neural Computation from CMU.
Research opportunities: I am looking to hire students at all levels (post-docs, PhDs, masters, undergrads, and visitors). I am also happy to collaborate and answer questions about my research and MIT academic programs. If you are interested, please send me an email. I especially encourage students from underrepresented groups to reach out.
News
- 2023: Excited to release some recent work formalizing and quantifying multimodal interactions from statistical (NeurIPS 2023) and human (ICMI 2023) perspectives, with applications in visualizing and interpreting multimodal models (ICLR 2023), contrastive learning of unique information (NeurIPS 2023), and guarantees for multimodal semi-supervised learning (ICLR 2024).
- 2023: Co-teaching 11-777 Multimodal Machine Learning, Fall 2023, course content will be updated on the website.
- 2023: Tutorials on multimodal machine learning at ICML 2023, ICMI 2023, CVPR 2022 and NAACL 2022, teaching at CIFAR DLRL summer school and African Masters of Machine Intelligence (day1, day2, day3, day4): check out our survey paper, slides, and videos.
- 2023: Nothing has excited me more than collaborating with and advising great students. I've learned so much from them and I'm excited to watch them embark on their new research agendas - follow their work for more exciting new ideas! Yun Cheng -> PhD at Princeton, Rulin Shao -> PhD at UW, Xiang Fan -> PhD at UW, Jivat Neet -> PhD at Berkeley, Yiwei Lyu -> PhD at Michigan, Yuxin Xiao -> PhD at MIT, Peter Wu -> PhD at Berkeley, Dong Won Lee -> PhD at MIT, Terrance Liu -> PhD at CMU.
- 2023: LP and I are teaching 2 new graduate seminar courses: 11-866 Artificial Social Intelligence and 11-877 Advanced Topics in Multimodal Machine Learning.
- 2022: Check out course content for 11-777 Multimodal Machine Learning, Fall 2022, where LP and I have completely revamped the course content. Also check out fully recorded lecture videos and course content for 11-777 in Fall 2020.
- 2022: Are you working on multimodal tasks and can't decide on a model? Check out HighMMT (TMLR 2022), our attempt at a single multimodal model that can predict sentiment, emotion, humor, disease, robot pose, and more, as well as MultiBench (NeurIPS 2021) and MultiZoo (JMLR 2022), a large-scale benchmark for multimodal learning spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
- 2021: Extremely honored to have received a Facebook PhD Fellowship and a Center for Machine Learning and Health Fellowship to support my research in socially intelligent AI! For students applying for graduate fellowships, I uploaded my statement from the 2020 application.
- 2020: Check out the CMU Machine Learning Blog - new research and educational content every few weeks on ML research going on at CMU!
Selected Publications
(* denotes joint first-authors, see full list of publications here)
Foundations of multimodal machine learning:
- HEMM: Holistic Evaluation of Multimodal Foundation Models
Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdinov, Louis-Philippe Morency
[arXiv] [code]
- Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2024
[arXiv] [code]
- Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2023
[arXiv] [code]
- Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
ACM Computing Surveys, Tutorials at ICML 2023, ICMI 2023, CVPR 2022, NAACL 2022
[arXiv] [tutorial website] [tutorial videos]
Multimodal AI for real-world sensors:
- IoT-LM: Large Multisensory Language Models for the Internet of Things
Shentong Mo, Ruslan Salakhutdinov, Louis-Philippe Morency, Paul Pu Liang
[arXiv] [code]
- MultiIoT: Benchmarking Machine Learning for the Internet of Things
Shentong Mo, Louis-Philippe Morency, Ruslan Salakhutdinov, Paul Pu Liang
[arXiv] [code]
Representation learning over multisensory and temporal data:
- High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov
TMLR 2022
[arXiv] [code]
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2021, JMLR Open Source Software 2022
[arXiv] [software] [website] [code]
- Multimodal Transformer for Unaligned Multimodal Language Sequences
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
ACL 2019
[arXiv] [code]
Socially-intelligent AI:
- Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions
Leena Mathur, Paul Pu Liang, Louis-Philippe Morency
[arXiv] [code]
- Computational Modeling of Human Multimodal Language: The MOSEI Dataset and Interpretable Dynamic Fusion
Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
Master's Thesis, CMU Machine Learning Data Analysis Project 2018 (first runner-up award)
[paper] [slides] [poster]
- Multimodal Sentiment Analysis with Word-level Fusion and Reinforcement Learning
Minghai Chen*, Sen Wang*, Paul Pu Liang*, Tadas Baltrušaitis, Amir Zadeh, Louis-Philippe Morency
ICMI 2017 (oral, honorable mention award)
[arXiv] [code] [slides]
Multimodal applications in health and wellness:
- Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction
Guillaume Jaume, Anurag Vaidya, Richard Chen, Drew Williamson, Paul Pu Liang, Faisal Mahmood
CVPR 2024
[arXiv] [code]
- Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Paul Pu Liang*, Terrance Liu*, Anna Cai, Michal Muszynski, Ryo Ishii, Nick Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2021 (oral)
[arXiv]
Real-world human-AI interaction:
- Towards Understanding and Mitigating Social Biases in Language Models
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, Ruslan Salakhutdinov
ICML 2021
[arXiv] [code]
- Towards Debiasing Sentence Representations
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2020
[arXiv] [code]
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2019 Workshop on Federated Learning (oral, distinguished student paper award)
[arXiv] [code]
Teaching
- Co-Instructor: 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2024, CMU with Daniel Fried
- Co-Lecturer: 11-777 Multimodal Machine Learning, Fall 2023, CMU with Louis-Philippe Morency
- Instructor: Multimodal Artificial Intelligence (day1, day2, day3, day4), African Masters Of Machine Intelligence, Summer 2023
- Co-Instructor: Tutorials on Multimodal ML at ICML 2023, ICMI 2023, CVPR 2022 and NAACL 2022 with Louis-Philippe Morency
- Co-Instructor: 11-866 Artificial Social Intelligence, Spring 2023, CMU with Louis-Philippe Morency
- Co-Instructor: 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2023, CMU with Louis-Philippe Morency
- Co-Lecturer: 11-777 Multimodal Machine Learning, Fall 2022, CMU with Louis-Philippe Morency
- Co-Instructor: 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2022, CMU with Louis-Philippe Morency and Amir Zadeh
- Guest Lecturer: 10-707 Deep Learning, 05-618 Human AI Interaction, 17-728 Machine Learning and Sensing, Peking University, University of Florida.
Lectures on multimodal machine learning [slides] [video]
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2020, CMU. Instructor: Louis-Philippe Morency
4 lectures on multimodal tasks [slides] [video], deep generative models [slides] [video], reinforcement learning [slides] [video], and multimodal RL [slides] [video].
Public videos on YouTube have amassed more than 10000 views.
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2019, CMU. Instructor: Louis-Philippe Morency
2 lectures on reinforcement learning [slides] and multimodal RL [slides]
- TA: 10-708 Probabilistic Graphical Models, Spring 2019, CMU. Instructor: Eric Xing
- TA: 10-715 Advanced Introduction to Machine Learning, Fall 2018, CMU. Instructor: Maria-Florina Balcan
- TA: 10-601 Introduction to Machine Learning, Fall 2016, CMU. Instructor: Roni Rosenfeld
- TA: 15-213/18-213/15-513 Introduction to Computer Systems, Summer 2016, CMU. Instructor: Brian Railing
Research Group
Some amazing students I've had the pleasure of advising:
- Haofei Yu, now PhD student at UIUC
- Rohan Pandey, now at Reworkd AI (YC S23) (best senior thesis award)
- Samuel Yu (CRA finalist)
- Yun Cheng, now PhD student at Princeton
- Rulin Shao, now PhD student at University of Washington
- Xiang Fan, now PhD student at University of Washington (CRA honorable mention)
- Jivat Neet, then research fellow at Microsoft Research, now PhD student at UC Berkeley
- Yiwei Lyu, now PhD student at University of Michigan (CRA honorable mention)
- Yuxin Xiao, now PhD student at MIT
- Peter Wu, now PhD student at UC Berkeley
- Dong Won Lee, now PhD student at MIT
- Xiangru Tang, now PhD student at Yale
- Terrance Liu, now PhD student at CMU
- Seong Hyeon Park, now PhD student at KAIST
- Chengfeng Mao, now PhD student at MIT
- Ziyin Liu, then PhD student at University of Tokyo, now PostDoc at MIT
- Irene Li, now at SoundHound (CRA honorable mention)
I have an Erdős number of 3 (Paul Erdős → Giuseppe Melfi → Erik Cambria → Paul Pu Liang).
This page has been accessed at least 
times since Feb 8, 2018.