Paul Liang
email:  ppliang (at) mit (dot) edu
office:  Wiesner Building E15-392

CV | Bio | Google Scholar
Github | Twitter

Research, teaching, diversity statements

I am an Assistant Professor at the MIT Media Lab and MIT EECS, where I direct the Multisensory Intelligence research group.

In summer 2024, I was a visiting researcher in the AI, psychology, and neuroscience program at UC Berkeley's Simons Institute for the Theory of Computing. Previously, I received my Ph.D. from the Machine Learning Department at Carnegie Mellon University, advised by Louis-Philippe Morency and Ruslan Salakhutdinov.

Prospective students: I am hiring at all levels (post-docs, PhDs, masters, undergrads, and visitors). If you want to join MIT as a graduate student, please apply through the programs in Media Arts & Sciences or EECS, and mention my name in your application.
I'm also happy to collaborate and answer questions about my research and MIT programs, I especially encourage students from underrepresented groups to reach out.

Some recent talks:



Research Group

Our group studies the foundations of multisensory AI and its impact on the human experience, through three complementary thrusts:

(1) Foundations of multisensory AI: The science and engineering of AI systems that can learn and interact with the world through integrating diverse sensory channels.

(2) Enhancing human experiences: Designing interactive AI technologies to augment human capabilities and improve overall well-being.

(3) Real-world human-AI interaction: Quantifying and mitigating real-world societal concerns for responsible deployment.

Group
Ziyin Liu (Visiting researcher)
Yi Ren Fung (Visiting researcher)
Antonis Christou (MAS)
Chanakya Ekbote (MAS)
David Dai (MAS)
Devin Murphy (EECS MEng, co-advised with Wojciech Matusik)
Lily Chen (EECS MEng)
Jimin Lee (EECS MEng)
Peilin Chen (EECS MEng)
Adithya Balachandran (EECS MEng)
Nidhish Sagar (EECS MEng, co-advised with Richard Braatz)
Minseok Jung (IDSS MS, co-advised with Lalana Kagal)
Steven-Shine Chen (UROP)
Hengzhi Li (UROP)
Former students
Haofei Yu, now PhD student at UIUC
Rohan Pandey, now researcher at Open AI (best CMU senior thesis award)
Yun Cheng, now PhD student at Princeton
Rulin Shao, now PhD student at University of Washington
Xiang Fan, now PhD student at the University of Washington (CRA outstanding undergrad researcher honorable mention)
Jivat Neet, then research fellow at Microsoft Research, now PhD student at UC Berkeley
Yiwei Lyu, now PhD student at the University of Michigan (CRA outstanding undergrad researcher honorable mention)
Yuxin Xiao, now PhD student at MIT
Peter Wu, now PhD student at UC Berkeley
Dong Won Lee, now PhD student at MIT
Terrance Liu, now PhD student at CMU
Chengfeng Mao, now PhD student at MIT
Ziyin Liu, then PhD student at the University of Tokyo, now PostDoc at MIT

Teaching
Spring 2025: MIT How to AI (Almost) Anything (coming soon!)
Spring 2025: MIT Introduction to Machine Learning (coming soon!)
Spring 2024: CMU 11-877 Advanced Topics in Multimodal Machine Learning, with Daniel Fried
Fall 2023: CMU 11-777 Multimodal Machine Learning, with Louis-Philippe Morency
Summer 2023: African Masters Of Machine Intelligence course on Multimodal AI (day1, day2, day3, day4)
2022-2023: Tutorials on Multimodal ML at ICML, ICMI, CVPR, NAACL with Louis-Philippe Morency
Spring 2023: CMU 11-866 Artificial Social Intelligence, with Louis-Philippe Morency
Spring 2023: CMU 11-877 Advanced Topics in Multimodal Machine Learning, with Louis-Philippe Morency
Fall 2022: CMU 11-777 Multimodal Machine Learning, with Louis-Philippe Morency
Spring 2022: CMU 11-877 Advanced Topics in Multimodal Machine Learning, with Louis-Philippe Morency, Amir Zadeh

  Selected publications (see full list here and on Google Scholar)


sym

HEMM: Holistic Evaluation of Multimodal Foundation Models
Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2024

arXiv | code | abstract | bibtex

  @article{liang2024hemm,
  title={HEMM: Holistic Evaluation of Multimodal Foundation Models},
  author={Liang, Paul Pu and Goindani, Akshay and Chafekar, Talha and Mathur, Leena and Yu, Haofei and Salakhutdinov, Ruslan and Morency, Louis-Philippe},
  journal={arXiv preprint arXiv:2407.03418},
  year={2024}
}
sym

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
ACM Computing Surveys 2024, Tutorials at ICML & ICMI 2023, CVPR & NAACL 2022

arXiv | website | videos | abstract | bibtex

  @article{liang2024foundations,
  title={Foundations \& trends in multimodal machine learning: Principles, challenges, and open questions},
  author={Liang, Paul Pu and Zadeh, Amir and Morency, Louis-Philippe},
  journal={ACM Computing Surveys},
  volume={56},
  number={10},
  pages={1--42},
  year={2024},
  publisher={ACM New York, NY}
}
sym

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2023

arXiv | code | abstract | bibtex

  @article{liang2024quantifying,
  title={Quantifying \& modeling multimodal interactions: An information decomposition framework},
  author={Liang, Paul Pu and Cheng, Yun and Fan, Xiang and Ling, Chun Kai and Nie, Suzanne and Chen, Richard and Deng, Zihao and Allen, Nicholas and Auerbach, Randy and Mahmood, Faisal and others},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2024}
}
sym

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov
TMLR 2022

arXiv | code | abstract | bibtex

  @article{liang2022high,
  title={High-modality multimodal transformer: Quantifying modality \& interaction heterogeneity for high-modality representation learning},
  author={Liang, Paul Pu and Lyu, Yiwei and Fan, Xiang and Tsaw, Jeffrey and Liu, Yudong and Mo, Shentong and Yogatama, Dani and Morency, Louis-Philippe and Salakhutdinov, Ruslan},
  journal={arXiv preprint arXiv:2203.01311},
  year={2022}
}
sym

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2021, JMLR Open Source Software 2022

arXiv | website | code | abstract | bibtex

  @article{liang2021multibench,
  title={Multibench: Multiscale benchmarks for multimodal representation learning},
  author={Liang, Paul Pu and Lyu, Yiwei and Fan, Xiang and Wu, Zetian and Cheng, Yun and Wu, Jason and Chen, Leslie and Wu, Peter and Lee, Michelle A and Zhu, Yuke and others},
  journal={Advances in neural information processing systems},
  volume={2021},
  year={2021},
}
sym

Towards Understanding and Mitigating Social Biases in Language Models
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, Ruslan Salakhutdinov
ICML 2021

arXiv | code | abstract | bibtex

  @inproceedings{liang2021towards,
  title={Towards understanding and mitigating social biases in language models},
  author={Liang, Paul Pu and Wu, Chiyu and Morency, Louis-Philippe and Salakhutdinov, Ruslan},
  booktitle={International Conference on Machine Learning},
  pages={6565--6576},
  year={2021},
  organization={PMLR}
}
sym

Think Locally, Act Globally: Federated Learning with Local and Global Representations
Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2019 Workshop on Federated Learning (oral, distinguished student paper award)

arXiv | code | abstract | bibtex

  @article{liang2020think,
  title={Think locally, act globally: Federated learning with local and global representations},
  author={Liang, Paul Pu and Liu, Terrance and Ziyin, Liu and Allen, Nicholas B and Auerbach, Randy P and Brent, David and Salakhutdinov, Ruslan and Morency, Louis-Philippe},
  journal={arXiv preprint arXiv:2001.01523},
  year={2020}
}
sym

Multimodal Transformer for Unaligned Multimodal Language Sequences
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
ACL 2019

arXiv | code | abstract | bibtex

  @inproceedings{tsai2019multimodal,
  title={Multimodal Transformer for Unaligned Multimodal Language Sequences},
  author={Tsai, Yao-Hung Hubert and Bai, Shaojie and Liang, Paul Pu and Kolter, J Zico and Morency, Louis-Philippe and Salakhutdinov, Ruslan},
  booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  pages={6558--6569},
  year={2019}
}
sym

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency
ACL 2018 (oral)

arXiv | code | abstract | bibtex

  @inproceedings{zadeh2018multimodal,
  title={Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph},
  author={Zadeh, AmirAli Bagher and Liang, Paul Pu and Poria, Soujanya and Cambria, Erik and Morency, Louis-Philippe},
  booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={2236--2246},
  year={2018}
}

Modified version of template from here