Volume 40, Number 2, March 2023 IEEE Signal Processing Magazine

Citation preview

IEEE SIGNAL PROCESSING MAGAZINE

PHYSICS-DRIVEN MACHINE LEARNING



VOLUME 40 NUMBER 2 | MARCH 2023

Get Published in the New IEEE Open Journal of Signal Processing Submit a paper today to the premier new open access journal in signal processing.

In keeping with IEEE’s continued commitment to providing

Your research will also be exposed to 5 million unique

options supporting the needs of all authors, in 2020, IEEE

monthly users of the IEEE Xplore® Digital Library.

introduced the high-quality publication, the IEEE Open Journal

The high-quality IEEE Open Journal of Signal Processing will

of Signal Processing.

draw on IEEE’s expert technical community’s continued

In recognition of author funding difficulties during this

commitment to publishing the most highly cited content.

unprecedented time, the IEEE Signal Processing Society, for

The editor-in-chief is the distinguished Prof. Brendt

2022, is offering a reduced APC of USD$995 with no page

Wohlberg, who specializes in signal and image processing,

limits. (This offer cannot be combined with any other

inverse problems, and computational imaging.

discounts.)

The rapid peer-reviewed process targets a publication

We invite you to have your article peer-reviewed and published

time frame within 10-15 weeks for most accepted papers.

in the new journal. This is an exciting opportunity for your

This journal is fully open and compliant with funder

research to benefit from the high visibility and interest the

mandates, including Plan S.

journal will generate.

Submit your paper today! The high-quality IEEE Open Journal of Signal Processing launched in IEEEXplore ® in January 2020 and welcomes submissions of novel technical contributions.

Click here to learn more

Digital Object Identifier 10.1109/MSP.2023.3241814

Contents

Volume 40 | Number 2 | March 2023

SPECIAL SECTION

DEEP OPTICAL CODING DESIGN 75  IN COMPUTATIONAL IMAGING

Henry Arguello, Jorge Bacca, Hasindu Kariyawasam, Edwin Vargas, Miguel Marquez, Ramith Hettiarachchi, Hans Garcia, Kithmini Herath, Udith Haputhanthri, Balpreet Singh Ahluwalia, Peter So, Dushan N. Wadduwage, and Chamira U.S. Edussooriya

PHYSICS-DRIVEN MACHINE LEARNING FOR COMPUTATIONAL IMAGING: PART 2 FROM THE GUEST EDITORS

Bihan Wen, Saiprasad Ravishankar, Zhizhen Zhao, Raja Giryes, and Jong Chul Ye

HYSICS-/MODEL-BASED AND 89  PDATA-DRIVEN METHODS FOR LOWDOSE COMPUTED TOMOGRAPHY

18  PHYSICS-EMBEDDED

Wenjun Xia, Hongming Shan, Ge Wang, and Yi Zhang

MACHINE LEARNING FOR ELECTROMAGNETIC DATA IMAGING

IGH-DIMENSIONAL MR 101 HSPATIOSPECTRAL IMAGING BY

Rui Guo, Tianyao Huang, Maokun Li, Haiyang Zhang, and Yonina C. Eldar

ON THE COVER In Part 2 of our special issue on physics-driven machine learning, we continue a panorama on recent developments in physics-driven machine learning techniques that can be applied for computational imaging. Here, the focus is on applications with a large variety of imaging, including optical imaging, tomographic imaging, hyperspectral image unmixing, magnetic resonance imaging, electromagnetic imaging, and Terahetz computational imaging. COVER IMAGE: ©SHUTTERSTOCK.COM/BAIVECTOR

46  UNFOLDING-AIDED BOOTSTRAPPED PHASE RETRIEVAL IN OPTICAL IMAGING

v. 1.15

v. 3.3

v. 3.7

Samuel Pinilla, Kumar Vijay Mishra, Igor Shevkunov, Mojtaba Soltanalian, Vladimir Katkovnik, and Karen Egiazarian

env2 v. 3.1

v. 1.11

Yanjie Zhu, Jing Cheng, Zhuo-Xu Cui, Qingyong Zhu, Leslie Ying, and Dong Liang

HYSICS-DRIVEN SYNTHETIC 129 PDATA LEARNING FOR BIOMEDICAL MAGNETIC RESONANCE

Qinqin Yang, Zi Wang, Kunyuan Guo, Congbo Cai, and Xiaobo Qu Input Signal, x, on a Graph, W or WN 4

3 2

INTEGRATION OF PHYSICS-BASED AND DATA-DRIVEN MODELS FOR HYPERSPECTRAL IMAGE UNMIXING Jie Chen, Min Zhao, Xiuheng Wang, Cédric Richard, and Susanto Rahardja

7

8 Convolutional Layer wo lT ne an Ch

61 

6

5

1

v. 2.9

PG. 141

QUANTITATIVE MAGNETIC RESONANCE IMAGING

e

env1

HYSICS-DRIVEN DEEP LEARNING 116 PMETHODS FOR FAST

On

Weng-Tai Su, Yi-Chun Hung, Po-Jen Yu, Chia-Wen Lin, and Shang-Hua Yang

Fan Lam, Xi Peng, and Zhi-Pei Liang

w1(0)

el

PHYSICS-GUIDED TERAHERTZ COMPUTATIONAL IMAGING

nn

32 

INTEGRATING PHYSICS-BASED MODELING AND DATA-DRIVEN MACHINE LEARNING

Ch a

13 

w1(1) w2(0)

y1 = w1(0)x + w1(1)WNx

w2(1)

y2 = w2(0)x + w2(1)WNx

o1 = ReLU{y1 + b1}

o2 = ReLU{y2 + b2} FC Layer vp(m) m = 1, 2, . . ., 16 p = 1, 2 p2

p1 SoftMax Output

PG. 155

IEEE SIGNAL PROCESSING MAGAZINE  (ISSN 1053-5888) (ISPREG) is published bimonthly by the Institute of Electrical and Electronics Engineers, Inc., 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA (+1 212 419 7900). Responsibility for the contents rests upon the authors and not the IEEE, the Society, or its members. Annual member subscriptions included in Society fee. Nonmember subscriptions available upon request. Individual copies: IEEE Members US$20.00 (first copy only), nonmembers US$248 per copy. Copyright and Reprint Permissions: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of U.S. Copyright Law for private use of patrons: 1) those post-1977 articles that carry a code at the bottom of the first page, provided the per-copy fee is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without fee. Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For all other copying, reprint, or republication permission, write to IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854 USA. Copyright © 2023 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Periodicals postage paid at New York, NY, and at additional mailing offices. Printed in the U.S.A. Postmaster: Send address changes to IEEE Signal Processing Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ 08854 USA. Canadian GST #125634188 

Digital Object Identifier 10.1109/MSP.2022.3219744

IEEESIGNAL SIGNALPROCESSING PROCESSINGMAGAZINE MAGAZINE IEEE

||

March 2023 January 2023

||

1

COLUMNS

IEEE Signal Processing Magazine EDITOR-IN-CHIEF

Christian Jutten—Université Grenoble Alpes, France

7 Society News

Boston Chapter Receives the 2022 Chapter of the Year Award!

AREA EDITORS

2022 IEEE Signal Processing Society Awards

Feature Articles Laure Blanc-Féraud—Université Côte d’Azur, France

141 Tips & Tricks

A Guide to Computational Reproducibility in Signal Processing and Machine Learning Joseph Shenouda and Waheed U. Bajwa

Special Issues Xiaoxiang Zhu—German Aerospace Center, Germany

Update on the CIC Multistage Decimation Filter With a Minimum Number of Additions per Output Sample (APOS): Can We Still Decrease the Number of APOS? Gordana Jovanovic Dolecek

Columns and Forum Rodrigo Capobianco Guido—São Paulo State University (UNESP), Brazil H. Vicky Zhao—Tsinghua University, R.P. China e-Newsletter Hamid Palangi—Microsoft Research Lab (AI), USA

155 Lecture Notes

Understanding the Basis of Graph Convolutional Neural Networks via an Intuitive Matched Filtering Approach Ljubiša Stankovic´ and Danilo P. Mandic

Social Media and Outreach Emil Björnson—KTH Royal Institute of Technology, Sweden

Explainable Machine Learning Manish Narwaria

EDITORIAL BOARD

DEPARTMENTS 3 From the Editor

Open and Reproducible Science: Desirable or Even Mandatory, But Not So Simple! Christian Jutten

4 President’s Message

Reaching Out to Members in the Middle East and India Athina Petropulu

Cover 3

©SHUTTERSTOCK.COM/STEFANO PANZERI

Cover 3 Dates Ahead

Massoud Babaie-Zadeh—Sharif University of Technology, Iran Waheed U. Bajwa—Rutgers University, USA Caroline Chaux—French Center of National Research, France Mark Coates—McGill University, Canada Laura Cottatellucci—Friedrich-Alexander University of Erlangen-Nuremberg, Germany Davide Dardari—University of Bologna, Italy Mario Figueiredo—Instituto Superior Técnico, University of Lisbon, Portugal Sharon Gannot—Bar-Ilan University, Israel Yifan Gong—Microsoft Corporation, USA Rémi Gribonval—Inria Lyon, France Joseph Guerci—Information Systems Laboratories, Inc., USA Ian Jermyn—Durham University, U.K. Ulugbek S. Kamilov—Washington University, USA Patrick Le Callet—University of Nantes,   France Sanghoon Lee—Yonsei University, Korea Danilo Mandic—Imperial College London, U.K. Michalis Matthaiou—Queen’s University Belfast, U.K. Phillip A. Regalia—U.S. National Science Foundation, USA Gaël Richard—Télécom Paris, Institut Polytechnique de Paris, France Reza Sameni—Emory University, USA Ervin Sejdic—University of Pittsburgh, USA Dimitri Van De Ville—Ecole Polytechnique Fédérale de Lausanne, Switzerland Henk Wymeersch—Chalmers University of Technology, Sweden

The IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) will be held in Rhodes Island, Greece, 4–9 June 2023.

ASSOCIATE EDITORS—COLUMNS AND FORUM Ulisses Braga-Neto—Texas A&M University, USA Cagatay Candan—Middle East Technical University, Turkey Wei Hu—Peking University, China Andres Kwasinski—Rochester Institute of Technology, USA Xingyu Li—University of Alberta, Edmonton, Alberta, Canada Xin Liao—Hunan University, China Piya Pal—University of California San Diego, USA Hemant Patil—Dhirubhai Ambani Institute of Information and Communication Technology, India Christian Ritz—University of Wollongong, Australia

ASSOCIATE EDITORS—e-NEWSLETTER Abhishek Appaji—College of Engineering, India Subhro Das—MIT-IBM Watson AI Lab, IBM Research, USA Behnaz Ghoraani—Florida Atlantic University, USA Panagiotis Markopoulos—The University of Texas at San Antonio, USA

IEEE SIGNAL PROCESSING SOCIETY Athina Petropulu—President Min Wu—President-Elect Ana Isabel Pérez-Neira—Vice President, Conferences Shrikanth Narayanan—VP Education K.V.S. Hari—Vice President, Membership Marc Moonen—Vice President, Publications Alle-Jan van der Veen—Vice President, Technical Directions

IEEE SIGNAL PROCESSING SOCIETY STAFF William Colacchio—Senior Manager, Publications and Education Strategy and Services Rebecca Wollman—Publications Administrator

IEEE PERIODICALS MAGAZINES DEPARTMENT Sharon Turk, Journals Production Manager Katie Sullivan, Senior Manager, Journals Production Janet Dudar, Senior Art Director Gail A. Schnitzer, Associate Art Director Theresa L. Smith, Production Coordinator Mark David, Director, Business Development Media & Advertising Felicia Spagnoli, Advertising Production Manager Peter M. Tuohy, Production Director Kevin Lisankie, Editorial Services Director Dawn M. Melley, Senior Director, Publishing Operations

Digital Object Identifier 10.1109/MSP.2022.3219742

SCOPE:  IEEE Signal Processing Magazine publishes tutorial-style articles on signal processing research and

IEEE prohibits discrimination, harassment, and bullying. For more information, visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

2

applications as well as columns and forums on issues of interest. Its coverage ranges from fundamental principles to practical implementation, reflecting the multidimensional facets of interests and concerns of the community. Its mission is to bring up-to-date, emerging, and active technical developments, issues, and events to the research, educational, and professional communities. It is also the main Society communication platform addressing important issues concerning all members.

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

FROM THE EDITOR Christian Jutten

| Editor-in-Chief | [email protected]

Open and Reproducible Science: Desirable or Even Mandatory, But Not So Simple!

I

n previous editorials, SPS President Athina Petropulu and I had the opportunity to say a few words about ethics, especially taking into account the usefulness of our research projects, for humanity and Earth, in a wide sense. In the current energy crisis and the explosion of costs, this issue becomes still more important, and I believe that it must be considered carefully in all our projects. Scientific integrity is another topic that I  often discuss as it is actually a duty for  all researchers for many of the reasons I developed in my November 2022 editorial [1]. In addition to ethics and scientific integrity, another important issue in research is related to open and reproducible science [2]. This implies that scientists must share datasets and codes when publishing new results. And reproducibility means that when using these data and codes and the attached information, any scientist should be able to reproduce the results, or to use the data and codes for conducting asfair-as-possible benchmarks. Open and reproducible science is essential for ensuring confidence in scientific results, and more widely to help society feel more confident in science and scientists. But reproducibility is not evident. In a Nature article [3], Baker writes: “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experi-

Digital Object Identifier 10.1109/MSP.2023.3235560 Date of current version: 17 February 2023

ments.” Those are some of the telling figures that emerged from Nature’s survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research. In this issue of IEEE Signal Processing Magazine (SPM), Shenouda and Bajwa [A1] address the issue of reproducibility from a practical point of view and provide a set of recommendations for sharing data and codes efficiently for the purpose of reproducible research. Of course, this implies that you must first be able to reproduce your own results. This article explains the main pitfalls to achieving reproducible experiments and then provides common tools and techniques that can be used to overcome each of those pitfalls, bearing in mind that making experiments reproducible can entail extra effort that may divert attention away from our primary research task. But, in addition to the issues discussed in this article, I believe that we must be aware of other replicability problems related to software and hardware architectures. The question of uncertainty in computing has been studied by computer scientists, explaining differences that can be obtained even for very simple codes when changing software versions, or when changing the hardware (e.g., running on a 32- or 64-bit processor) [4]. It has also been shown that these uncertainties are not independent. For instance, compile-time and runtime options can interplay on performance [5]. In many domains, e.g., in neuroimaging, datasets and codes are shared IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

by scientists from many countries. But processing the same data with different pipelines, or even with the same pipeline, by changing a few parameters (even just one), can provide very different results [6]. Other results show changes can appear with different operating systems, software packages, or workstation types [7], [8]. Because I think that these issues deserve to be published in SPM, I invited scientists working on computational uncertainty to write a tutorial for IEEE Signal Processing Society members.

In this issue This issue of SPM is primarily comprised of the second part of the special issue on “Physics-Driven Machine Learning for Computational Imaging,” with nine articles detailed in [A2]. These articles consider a large variety of imaging, including optical imaging, tomographic imaging, hyperspectral image unmixing, magnetic resonance imaging, electromagnetic imaging, and terahertz computational imaging. Although the physics behind these various technics is very different, the main message to take home is the impact of physics-driven learning for designing simpler, faster, and more explainable methods, or the ability for providing, when few data are available, physics-driven synthetic data, which are relevant. This issue also contains three column and forum articles in addition to [A1]. In [A3], Dolecek explores some (continued on page 12) 3

PRESIDENT’S MESSAGE Athina Petropulu

| IEEE Signal Processing Society President | [email protected]

Reaching Out to Members in the Middle East and India

A

s I am writing this article, I am wrapping up a trip as IEEE Signal Processing Society (SPS) president to Doha, Qatar (9–11 January), to speak at the 2022 IEEE Spoken Language Technology (SLT) Workshop, and India (12–16 January), for technical talks and meetings with local signal processing researchers and SPS local Chapter chairs. Speech and language processing deals with speech recognition, text-to-speech synthesis, spoken language understanding, speech-to-speech translation, spoken dialog management, and many other areas that are relevant to a wide range of applications. The IEEE Speech and Language Processing Technical Committee (SLTC) is a vibrant segment of the SPS, encompassing the most active and accomplished researchers and technologists in the field. The SLT Workshop is a flagship event of the SLTC, bringing together researchers from academia and industry to discuss cutting-edge developments in automatic speech recognition and understanding. The 2022 SLT was held 9–12 January 2023 and was the first speech conference held in the Middle East and the first speech conference to be held in an Arabic-speaking nation. It was organized by well-respected leaders of industry and academia, with Dr. Ahmed Ali, Qatar Computing Research Institute, and Dr. Bhuvana Ramabhadran, Google, as general chairs. Among the many firsts of the 2022 SLT was the SLT Code Hackathon

Digital Object Identifier 10.1109/MSP.2023.3238232 Date of current version: 17 February 2023

4

on the topic of low-resource speech and language technology and applications. The SLT Code Hackathon was open to anyone interested in speech and language technology, including researchers, programmers, user experience designers, practitioners, and enthusiasts. Inperson participation was preferred, but remote participation was possible if at least one team member was local. Teams had three to six members. The 2022 hackathon chairs were Thomas Schaaf (3M M*Modal), Gianni Di Caro (Carnegie Mellon University in Qatar), Shinji Watanabe (Carnegie Mellon University), Paola Garcia (Johns Hopkins University), Mirco Ravanelli (Université de Montréal), Alessandra Cervone (Amazon), Mus’ab Husaini (Qatar Computing Research Institute), Harshita Diddee (Microsoft). This was a very successful event that attracted 97 participants from over 30 countries. Eighteen teams presented projects on the recognition of spoken dialects in different parts of the world (e.g., Africa and India) that are not well captured by existing speech and language technology tools. The projects were evaluated for their potential impact (social, research, and business), their degree of innovation, and how well they were implemented. I had the pleasure of serving as a judge and thoroughly enjoyed the over 3 h of presentations by very enthusiastic teams of students, from high-school students to Ph.D. students. It was amazing to see the energy and the talent of those students from all parts of the world. Kudos to the SLT chairs and the hackathon organizers, who conceived the hackathon idea and inIEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

vested great effort in making this event a big success. Hackathons, challenges, and competitions generate a lot of enthusiasm and are great vehicles to attract talent to the field of signal processing. Encouraging and supporting such events are among the priorities of the SPS. I also gave a talk at the 2022 SLT opening ceremony about the state of the Society, reinforcing the strong commitment of the SPS to serve SLT researchers and technologists. While in Doha, I also visited various universities and met with colleagues working on signal processing-related topics, hosted by Prof. Erchin Serpedin. I gave technical talks at the Department of Electrical and Computer Engineering (ECE) of Texas A&M University Qatar (TAMU-Q) and Qatar University (QU) and interacted with many colleagues. I also gave a lecture to a capstone class on IEEE ethics and how one should be mindful of the effects of technology on people’s safety and privacy, the propagation of stereotypes that hold women and underrepresented minorities back, and consequences for the environment. I had the honor of meeting with the dean of engineering of Hamad Bin Khalifa University (HBKU) as well as with several HBKU faculty, the vice president of research at Qatar University, and the president and the associate provost for academic affairs of the University of Doha for Science and Technology (UDST). I was given extensive tours of their facilities. I am thankful to all for their exceptional hospitality and was impressed with their efforts to stay at the forefront of research and education.

The 2022 SLT Hackathon participants, organizers, and judges.

Over the past 10 years, Doha developed a large state-of-art infrastructure to host the 2022 soccer World Cup. Many colleagues in Qatar are eager to leverage that infrastructure and be more involved in SPS conference organization in the future. The SPS membership in Qatar is not very high, definitely not as high as that of the IEEE Communication Society. We discussed the strong focus of the SPS on physical layer communications and how, with its conferences, publications, and educational materials, the SPS can help researchers understand the unique challenges that next-generation communication systems face at the physical layer. I certainly hope that we will see more member engagement in the future. Several colleagues expressed interest in the new SPS-curated educational content that provides certificates. My India trip was filled with talks and meetings with SPS members and volunteers from academia and industry. Prof. K.V.S. Hari, past SPS vice president of membership, did a fantastic job in energizing SPS membership and SPS Chapters in India. He organized various activities to bring members together and meet with me during my visit. On 12 January, the SPS Bangalore Chapter hosted me at the IEEE SPS India Conclave, in Bangalore. I was pleasantly surprised to walk into a room of the IEEE India office, at the World Trade Center, Bangalore, full of SPS volunteers. Ten very enthusiastic SPS Chapter chairs gave me presentations on their Chapter activities and their plans for the future. It was a great session, which ended with a nice lunch at a nearby hotel. Many thanks to the IEEE India office team for all the arrangements. Later the same day, there was a panel discussion on “Advances in Signal Processing: Opportunities and Challenges.” The event was held in the Sarah Kailath Auditorium, which was built using the generous donation from Prof. Thomas

Kailath, Stanford University, a pioneering researcher, extraordinary educator, and technology leader in signal processing and other fields. The panelists included Santhosh Kumar [managing director, Texas Instruments (India)], Dr. Aloknath De (chair, IEEE Bangalore Section; former corporate vice president and chief technology officer, Samsung Research; and the panel moderator), Ravikiran Annaswamy (president, IEEE Technology and Engineering Management Society, and founder of Numocity, an electric vehicle technology company), Dr. Geetha Manjunath [founder of Niramai Healthcare Analytix, which has developed radiation-free thermal imaging-based breast cancer detection technology using artificial intelligence (AI)], and myself. The discussion touched upon various signal processing aspects of system design in medical imaging, electric vehicles, radar and communication systems, and next-generation factories, among other applications. The importance of sensing, signal processing, optimization tools, and incorporating domain knowledge into AI/machine learning algorithms was highlighted. The need for a “systems approach” to problem solving was emphasized, which also means that a fresh look at the engineering curriculum is needed. Bangalore is viewed as the Silicon Valley of India, and this event brought together a large number of industry representatives interested in signal processing who had to fight heavy traffic to reach the venue at the end of their working day. I was impressed with their strong interest to stay connected with the signal processing research community. Following the panel, I gave a presentation on the state of the SPS and the opportunities the SPS provides to members for their career development. On 16 January, a big gathering of K–12 students and their teachers was organized by Women in Signal Processing IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

Enjoying the hospitality of Qatar colleagues in Doha. From left to right: Dr. Rachid Benlamri (VP UDST), Dr. Athina Petropulu, Dr. Ali Ghrayeb (TAMU-Q), Dr. Joseph Boutros (TAMU-Q),Dr. Erchin Serpedin (ECE chair, TAMU-Q), Dr. Haithan Abu-Rub (TAMU-Q), Dr. Ridha Hamila (QU).

(WISP) at the Indian Institute of Science (IISc) in Bangalore. A WISP member interviewed me about my career path, why I chose signal processing as my research area, and what the potential career paths are for a student with signal processing experience. Later the same day, I gave the inaugural talk at the ECE–IISc– Distinguished Visitor Program on the topic of dual-function radar communication systems. The talk as well as the follow-up reception were very well attended by faculty, students, and alumni. During my visit in India, I also visited the Indian Institute of Technology Delhi, hosted by Prof. Monika Aggarwal, the IEEE SPS India Conclave, Delhi and the Women in Engineering Delhi Section, where I gave a technical talk and also interacted with many faculty and students. Following my talk, there was a poster session, where students presented work related to signal processing. During that time, we honored Prof. S.C. Dutta Roy, 2021 recipient of the SPS Regional Distinguished Teacher Award, which recognizes individuals who have excelled in the teaching of signal processing. With Prof. Aggarwal, we traveled to Agra, where we visited the Dayalbagh Educational 5

The panel discussion on “Advances in Signal Processing: Opportunities and Challenges” at the IEEE SPS India Conclave, Bangalore.

Institute (DEI), an open-air-classroom university focused on sustainable practices, self-discipline, and spirituality. DEI has invested in significant outreach to the neighboring low-income community. In Agra, we had the opportunity to experience the unparalleled beauty of the Taj Mahal. In all places, I was treated to outstanding hospitality, including great Indian food and warm social interactions. Everybody in India is extremely excited about ICASSP 2025, to be held in Hyderabad. The ICASSP 2025 machinery is already in motion, preparing to offer a great technical program and also welcome the SPS community in India. I am sure the ICASSP 2025 attendees will be in for a great treat. It has been a great pleasure and honor for me to be greeted with such warmth and enthusiasm by SPS members in Qatar and India. It was an opportunity to experience the great name recognition and respect SPS enjoys around the world and also to feel the strong bond that connects SPS members. We parted with the wish that we will meet again at the 2023 ICASSP, in Rhodes, Greece, this coming June!

The IEEE SPS India Conclave, Delhi.

IMAGE LICENSED BY GRAPHIC STOCK

We want to hear from you!

Acknowledgment Special thanks to Prof. Erchin Sepredin, chair of Electrical and Computer Engineering at TAMU-Q, for organizing my Qatar visit and Prof. K.V.S., IISc, and Prof. Monica Aggarwal, IIT Delhi, for organizing my India visit.

Do you like what you’re reading? Your feedback is important. Let us know—send the editor-in-chief an e-mail!



6

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

SP

SOCIETY NEWS

Boston Chapter Receives the 2022 Chapter of the Year Award!

T

he IEEE Signal Processing Society (SPS) Boston Chapter has been selected as the recipient of the 2022 Chapter of the Year Award. The Chapter of the Year Award will be presented at the ICASSP 2023 Awards Ceremony in Rhodes Island, Greece. The award is presented annually to Chapters that have provided their membership with the highest quality of programs, activities, and services. The SPS Boston Chapter will receive a plaque and a check in the amount of US$1,000 to support local Chapter activities. The Chapter will publish an article in a future issue of IEEE Inside Signal Processing eNewsletter.

44 SPS members elevated to IEEE Fellow Each year, the IEEE Board of Directors confers the grade of Fellow on up to one-tenth of one percent of the voting members. To qualify for consideration, an individual must have been a Member, normally for five years or more, and a Senior Member at the time of nomination to Fellow. The grade of Fellow recognizes unusual distinction in IEEE’s designated fields. The SPS congratulates the following 44 SPS members who were recognized with the grade of Fellow as of 1 January 2023. Digital Object Identifier 10.1109/MSP.2023.3236464 Date of current version: 17 February 2023

Farhan A. Baqai, for contributions in leadership in digital camera image processing. Alfred M. Bruckstein, for contributions to signal representation and swarm robotics. Carlos A. Busso, for contributions to speech and multimodal affective signal processing and their technology applications. Patrizio Campisi, for contributions to the development of biometrics. Constantine Caramanis, for contributions to robust statistics and optimization in high dimensions. Tsung-Hui Chang, for contributions to distributed optimization methods and their applications in signal processing and wireless communications. Symeon Chatzinotas, for contributions to precoding technologies for multiple antennas. Yuejie Chi, for contributions to statistical signal processing with lowdimensional structures. Harpreet Dhillon, for contributions to heterogeneous cellular networks. Ayman El-Baz, for contributions to artificial intelligence (AI) in medicine. Peter Gerstoft, for contributions to environmental signal processing and geoacoustic array processing. Marios Kountouris, for contributions to optimization and multiantenna techniques in heterogeneous wireless ­networks. Zhengguo Li, for contributions to video encoding and streaming optimization and edge-preserving filters. IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

Wei Liu, for contributions to largescale machine learning and multimedia intelligence. Michail Matthaiou, for contributions to fundamental research and practical implementation of massive multipleinput, multiple-output (MIMO). Gerald Matz, for contributions to signal processing for communications in nonstationary environments. Florian E. Metze, for contributions to end-to-end training of speech recognition systems. Chunyan Miao, for contributions to multimodal signal processing and AI technologies for aging-at-home and population health. Ralf Reiner Müller, for contributions to the design and analysis of large multiantenna and multiple-access systems. Chandra R. Murthy, for contributions to Bayesian sparse signal recovery and energy-harvesting communications. Kazuhiro Nakadai, for contributions to robot audition and computational auditory scene analysis. Premkumar Natarajan, for contributions to conversational AI systems, spoken language translation, and home voice-assistant systems. Hideki Ochiai, for contributions to power and spectral-efficient wireless communication. John Pauly, for contributions to data acquisition and image reconstruction methods for magnetic resonance imaging. 7

Farhan A. Baqai

Alfred M. Bruckstein

Carlos A. Busso

Patrizio Campisi

Constantine Caramanis

Tsung-Hui Chang

Symeon Chatzinotas

Yuejie Chi

Harpreet Dhillon

Ayman El-Baz

Peter Gerstoft

Marios Kountouris

Zhengguo Li

Wei Liu

Michail Matthaiou

Gerald Matz

Florian E. Metze

Chunyan Miao

Ralf Reiner Müller

Chandra R. Murthy

Kazuhiro Nakadai

Premkumar Natarajan

Hideki Ochiai

John Pauly

Michael Polley

Daniel Povey

James Preisig

Miguel Raul D. Rodrigues

Congratulations!

SPS Members and Affiliates Elected IEEE Fellow Class 2023

Stephanie Schuckers

Gonzalo Seco-Granados

Xin Wang

Shinji Watanabe

Anthony Man-Cho So

Houbing Song

Stefan A. Werner Jason D. Williams

Yongxing Zhou

8

Changho Suh

Olav Tirkkonen

Xiaoyu Wang

Brendt Wohlberg

Shui Yu

Yao Zhao

Chengqing Zong

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

Michael Polley, for leadership in multimedia chipset architectures and mobile camera technologies. Daniel Povey, for contributions to acoustic modeling for speech recognition. James Preisig, for contributions to underwater acoustic communication channel modeling, signal processing, and performance prediction. Miguel Raul D. Rodrigues, for contributions to multimodal data processing and foundations of reliable and secure communications. Stephanie Schuckers, for contributions in biometric recognition systems. Gonzalo Seco-Granados, for contributions to signal processing for global navigation satellite systems, and 5G localization systems. Anthony Man-Cho So, for contributions to optimization in signal processing and communications.

Houbing Song, for contributions to big data analytics and integration of AI with the Internet of Things. Changho Suh, for contributions to interference management and distributed storage codes. Olav Tirkkonen, for contributions in the theory and practice of wireless communications technology and standards. Xiaoyu Wang, for contributions to video analysis technologies for embedded systems. Xin Wang, for outstanding contributions to wireless localization and dynamic resource allocation in broadband mobile networks. Shinji Watanabe, for contributions to speech recognition technology. Stefan A. Werner, for contributions to in-band full-duplex wireless communication systems and selective datareuse online learning.

Jason D. Williams, for contributions to the theory and practice of machinelearning-based spoken ­d ialogue systems. Brendt Wohlberg, for contributions to computational imaging and sparse representations. Shui Yu, for contributions to cybersecurity and privacy. Yao Zhao, for contributions to image/video analysis and multimedia content protection. Yongxing Zhou, for contributions to MIMO beamforming codebooks and smart spectrum access in wireless ­networks. Chengqing Zong, for contributions to machine translation and natural language processing.

2022 IEEE Signal Processing Society Awards

T

he IEEE Signal Processing Society (SPS) congratulates the following members who will receive the Society’s prestigious awards during ICASSP 2023.

The Norbert Wiener Society Award honors outstanding technical contributions in a field within the scope of the SPS and outstanding Richard Baraniuk leadership within that field. The Norbert Wiener Society Award includes a plaque, a certificate, and a monetary award of US$2,500. It is the highest-level award bestowed by the SPS. This year’s recipient is Digital Object Identifier 10.1109/MSP.2023.3236465 Date of current version: 17 February 2023

Richard Baraniuk, “for fundamental contributions to sparsity-based signal processing and pioneering broad dissemination of open educational resources.” The Claude Shannon– Harry Nyquist Technical Achievement Award honors a person who, over a period of years, has made outstanding Nicholas technical contributions Sidiropoulos to theory and/or practice in technical areas within the scope of the Society, as demonstrated by publications, patents, and a recognized impact in this field. The prize for the award is US$1,500, a plaque, and a certificate. The recipients of the Claude IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

Shannon–Harry Nyquist Technical Achievement Award are Nicholas Sidiropoulos, “for exemplary contributions to tensor decomposition, Arnold Lee beamforming, and Swindlehurst spectral analysis” and Arnold Lee Swindlehurst, “for contributions to multiuser and multiantenna com­munications and sensor array signal processing.” The Carl Friedrich Gauss Education Award honors educators who have made pioneering and significant contributions to signal H. Vincent Poor processing education. 9

Judging is based on a career of meritorious achievement in signal processing education as exemplified by the writing of scholarly books and texts, course materials, and papers about education; inspirational and innovative teaching; creativity in the development of new curricula and methodology. The award comprises a plaque, a monetary award of US$1,500, and a certificate. The recipient of the Signal Processing Society Carl Friedrich Gauss Education Award is H. Vincent Poor, “for outstanding contributions to education and mentoring in statistical signal processing and wireless communications.”

Ahmed Tewfik

Tulay Adali

The Leo L. Beranek Meritorious Service Award was presented this year to Ahmed Tewfik and Tulay Adali, “for exemplary service to and leadership in the Signal Processing Society.” The award comprises a plaque and a certificate; judging is based on dedication, effort, and contributions to the Society.

The Amar G. Bose Ind­ ustrial Leader Award recognizes an industry business or technical leader whose leadership has resulted in Xuedong Huang major and outstanding adva­nces or new directions using signal processing technologies within the scope of the Society. This award is for executive leadership resulting in major advances and new directions using signal processing in a business a rea. T he pr ize is US$1,50 0, a plaque, and a certificate. The recipient of the Amar G. Bose Industrial Leader Award is Xuedong Huang, “for contributions to speech recognition and industrial leadership in artificial intelligence.”

10

The Industrial Innovation Award is presented this year to Ivan Tashev, “for outstanding contributions to microphone array and speech Ivan Tashev enh­ancement systems.” The Industrial Innovation Award recognizes an individual or team at any level who were industry employees whose technical contributions have resulted in significant advances using signal processing technologies within the scope of the Society. Selection is based on major industrial accomplishments, standards, deployment of important processes or products, and so on that are of substantial benefit to the public, use signal processing technologies, and are visible beyond the company or institution where the contribution was made. The award is open to individuals at any level who were industry employees who played a significant role in the technical contribution at the time of the accomplishments being recognized. The prize includes US$1,500 per awardee (up to a maximum of US$4,500 per award), a plaque, and a certificate. The Pierre-Simon Laplace Early Career Technical Achievement Award honors an individual who, over a period of years in his/ Mingyi Hong her early career, made significant technical contributions to theory and/or practice in technical areas within the scope of the Society, as demonstrated by publications, patents, and a recognized impact on the field, including but not limited to a standard, product, or technology trend. The award comprises a plaque, a monetary award of US$1,500, and a certificate. The recipient of the Pierre-Simon Laplace Early Career Technical Achievement Award is Mingyi Hong, “for contributions to nonconvex, distributed and learning-based optimization for signal processing.”

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

Elvin Isufi

Geethu Joseph

The Best Ph.D. Dissertation Award recognizes relevant signal processing doctoral work that stimulates further research in the field. The award consists of a monetary award of US$1,500 and a certificate. The recipients of the Best Ph.D. Dissertation Award are Elvin Isufi and Geethu Joseph.

The Meritorious Regional/Chapter Service Award honors the outstanding contributions of any member of the Society to regional Sarath S activities of the SPS. Judging is based on dedication, effort, and contributions made to activities aimed at promoting the technical and educational activities of the SPS in one specific Region/Chapter as well as its local membership participation. The award comprises a plaque and a certificate. The recipient of the Meritorious Regional/Chapter Service Award is Sarath S., “for leadership and outstanding contributions as a volunteer and mentor at Section and Regional levels.” Six Best Paper Awards were awarded, honoring the author(s) of a paper of exceptional merit dealing with a subject related to the Society’s technical scope and appearing in one of the Society’s transactions, irrespective of the author’s age. The prize is US$500 per author (up to a maximum of US$1,500 per award) and a certificate. Eligibility is based on a six-year window preceding the year of election, and judging is based on general quality, originality, subject matter, and timeliness. Up to six Best Paper Awards may be presented each year. This year, the awardees are ■■ Morten Kolbæk, Dong Yu, ZhengHua Tan, and Jesper Jensen, “Multitalker Speech Separation With Utterance-Level Permutation

I nva r i a n t Tr a i n i n g o f D e e p Recurrent Neural Networks,” IEEE/ ACM Transactions on Audio, Speech, and Language Processing, October 2017. ■■ Haoran Sun, Xiangyi Chen, Qingjiang Shi, Mingyi Hong, Xiao Fu, and Nicholas D. Sidiropoulos, “Learning to Optimize: Training Deep Neural Networks for In­­ terference Management,” IEEE Transactions on Signal Processing, October 2018. ■■ Fernando Gama, Joan Bruna, and Alejandro Ribeiro for “Stability Pro­ perties of Graph Neural Networks”, IEEE Transactions on Signal Proc­ essing, September 2020. ■■ Stanley H. Chan, Xiran Wang, and Omar A. Elgendy, “Plug-and-Play ADMM for Image Restoration: FixedPoint Convergence and Applications,” IEEE Transactions on Computational Imaging, January 2017. ■■ David I Shuman, Pierre Vandergheynst, Daniel Kressner, and Pascal Frossard, “Distributed Signal Processing via Chebyshev Polynomial Approxi­ mation,” IEEE Transactions on Signal and Information Processing over Networks,” December 2018. ■■ Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farokhi Farhad, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor, “Federated Learning With Differential Privacy: Algorithms and Performance Analysis,” IEEE Transactions on Information Forensics and Security, April 2020. The Donald G. Fink Overview Paper Award honors the author(s) of a journal article of broad interest that has had substantial impact over several years on a subject related to the Society’s technical scope. A paper considered for the award should present an overview of a method or theory with technical depth and application perspective. It should have a multiyear record of impact and also be relevant to current researchers and/or practitioners. The prize consists of US$500 per author (up to a maximum of US$1,500 per award) and a certificate. This year, the Donald G. Fink

Overview Paper Award recipients are Nicholas D. Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E. Papalexakis, and Christos Faloutsos, for “Tensor Decomposition for Signal Processing and Machine Learning,” IEEE Transactions on Signal Processing, July 2017. The IEEE Signal Processing Letters Best Paper Award honors the author(s) of a letter article of exceptional merit and broad interest on a subject related to the Society’s technical scope and appearing in IEEE Signal Processing Letters. The prize consists of US$500 per author (up to a maximum of US$1,500 per award) and a certificate. To be eligible for consideration, an article must have appeared in IEEE Signal Processing Letters in an issue predating the Spring Awards Board meeting by five years (typically held in conjunction with ICASSP). Judging is based on technical novelty, research significance, and the quality and effectiveness in presenting subjects in an area of high impact to the Society’s members. The recipient of the IEEE Signal Processing Letters Best Paper Award is Lorenzo Vangelista, for “Frequency Shift Chirp Modulation: The LoRa Modulation,” IEEE Signal Processing Letters, December 2017. The IEEE Signal Processing Magazine Best Column Award honors the author(s) of a column of exceptional merit and broad interest on a subject related to the Society’s technical scope and appearing in the Society’s magazine. The prize consists of US$500 per author (up to a maximum of US$1,500 per award) and a certificate. In the event that there are more than three authors, the maximum prize is divided equally among all authors, each of whom receives a certificate. This year, the IEEE Signal Processing Magazine Best Column Award recipients are Dong Yu and Li Deng, for the article “Deep Learning and Its Applications to Signal and Information Processing [Exploratory DSP],” published in the January 2011 issue of IEEE Signal Processing Magazine. The IEEE Signal Processing Magazine Best Paper Award honors the

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

author(s) of an article of exceptional merit and broad interest on a subject related to the Society’s technical scope and appearing in the Society’s magazine. The prize includes US$500 per author (up to a maximum of US$1,500 per award) and a certificate. In the event that there are more than three authors, the maximum prize is divided equally among all authors, each of whom receives a certificate. This year, the IEEE Signal Processing Magazine Best Paper Award recipients are Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdelrahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury, for the article “Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups,” published in the November 2012 issue of IEEE Signal Processing Magazine. The Sustained Impact Paper Award honors the author(s) of a journal article of broad interest that has had sustained impact over many years on a subject related to the Society’s technical scope. The prize consists of US$500 per author (up to a maximum of US$1,500 per award) and a certificate. In the event that there are more than three authors, the maximum prize is divided equally among all authors, each of whom receives a certificate. To be eligible for consideration, an article must have appeared in one of the SPS transactions or in Journal of Selected Topics in Signal Processing in an issue predating the Spring Awards Board meeting by at least 10 years (typically held in conjunction with ICASSP). This year, the Sustained Impact Paper Award recipients are Petre Stoica and Arye Nehorai, for “MUSIC, Maximum Likelihood, and Cramér–Rao Bound,” published in IEEE Transactions on Acoustics, Speech, and Signal Processing, May 1989. The Young Author Best Paper Award honors the author(s) of an especially meritorious paper dealing with a subject related to the Society’s technical scope and appearing in one of the Society’s transactions and who, upon date of submission of the paper, is younger

11

than 30. The prize is US$500 per author (up to a maximum of US$1,500 per award) and a certificate. Eligibility is based on a four-year window preceding the year of election, and judging is based on general quality, originality, subject matter, and timeliness. Two Young Author Best Paper Awards are being presented this year: ■■ Ashutosh Pandey for the paper coauthored with DeLiang Wang, “A New Framework for CNN-Based Speech Enhancement in the Time Domain,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, July 2019. ■■ Qiuqiang Kong and Turab Iqbal for the paper coauthored with Yin Cao, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley, “PANNs: LargeScale Pretrained Audio Neural

FROM THE EDITOR 

SPS members receive 2023 IEEE awards The IEEE James L. Flanagan Speech and Audio Processing Technical Field Award will be presented to Alexander Waibel, “for pioneering contributions to spoken language translation and supporting technologies.” The IEEE Fourier Award for Signal Processing will be presented to Rabab Kreidieh Ward, “for outstanding contributions to advancing signal processing techniques and their practical applications, and for technical leadership.” IEEE has announced the recipients of the 2023 IEEE medals, which are

the highest-honor awards it presents. The medals will be given at the 2023 IEEE Honors Ceremony. Two SPS members have been awarded an IEEE medal for 2023: ■■ The IEEE Jack S. Kilby Signal Processing Medal for outstanding achievements in signal processing will be presented to José M.F. Moura, “for contributions to theory and practice of statistical, graph, and distributed signal processing.” ■■ The IEEE James H. Mulligan, Jr. Education Medal for outstanding contributions to education will be presented to James J. Truchard, “for the development of LabVIEW and establishing worldwide programs to enhance hands-on learning in laboratories and classrooms.” SP 

(continued from page 3)

tips and tricks to decrease the number of additions per output sample in a cascaded integrator-comb multistage decimation filter. Two “Lecture Notes” focus on simple signal processing examples for understanding graph convolutional neaural networks [A4] and making more explainable deep learning [A5]. Although these two articles use examples related to a simple linear filtering, for which we can wonder, what is the interest in using a nonlinear model, I think that these articles are interesting from a didactic point of view. Especially, in [A5], the same data (related to a twoor three-taps filter) are trained with four different neural architectures, all very simple. Although after training the different architectures achieve good fit of the filter, the explainability is not possible despite the network simplicity. Due to the black-box nature of the networks, even simple (six weights and three neurons for three of them), discussion clearly shows the impossibility of relating the weights of the network to the physical parameters of the filter. In the last part, the author suggests what is called a system-centric philosophy, which, in fact, suggests the use of some steps based on prior 12

Networks for Audio Pattern Recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, October 2020.

knowledge of the system to learn. This is exactly the same philosophy as the one supported in all the articles of the special issue on “Physics-Driven Machine Learning for Computational Imaging.” I wish everyone an enjoyable and rewarding read.

Appendix: Related Articles [A1] J. Shenouda and W. U. Bajwa, “A guide to computational reproducibility in signal processing and machine learning,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 141–151, Mar. 2023, doi: 10.1109/MSP.2022.3217659. [A2] B. Wen, S. Ravishankar, Z. Zhao, R. Giryes, and J. C. Ye, “Physics-driven machine learning for computational imaging: Part 2,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 13–15, Mar. 2023, doi: 10.1109/MSP.2023.3236492. [A3] G. J. Dolecek, “Update on the CIC multistage decimation filter with a minimum number of additions per output sample (APOS): Can we still decrease the number of APOS?” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 151–154, Mar. 2023, doi: 10.1109/MSP.2022.3216720. [A4] L. Stankovic´ and D. Mandic, “Understanding the basis of graph convolutional neural networks via an intuitive matched filtering approach,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 155–165, Mar. 2023, doi: 10.1109/MSP.2022.3207304. [A5] M. Narwaria, “Explainable machine learning – The importance of a system-centric perspective,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 165– 172, Mar. 2023, doi: 10.1109/MSP.2022.3211368.

References

[1] C. Jutten, “Scientific integrity: A duty for researchers [From the Editor],” IEEE Signal Process.

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

Mag., vol. 39, no. 6, pp. 3–84, Nov. 2022, doi: 10.1109/MSP.2022.3198298. [2] National Academies of Sciences, Engineering, and Medicine et al., Reproducibility and Replicability in Science. Washington, DC, USA: National Academy Press, 2019. [Online]. Available: https://www.ncbi. nlm.nih.gov/books/NBK547532/ [3] M. Baker, “1,500 scientists lift the lid on reproducibility,” Nature, vol. 533, no. 7604, pp. 452–454, May 2016, doi: 10.1038/533452a. [4] S. Bernardi, M. Famelis, J.-M. Jézéquel, R. Mirandola, D. Perez Palacin, F. A. C. Polack, and C. Trubiani, “Living with uncertainty in model-based development,” in Composing Model-Based Analysis Tools, R. Heinrich, F. Durán, C. Talcott, and S. Zschaler, Eds. Cham, Switzerland: Springer-Verlag, 2021, pp. 159–185. [5] L. Lesoil, M. Acher, X. TëRnava, A. Blouin, and J.-M. Jézéquel, “The interplay of compile-time and run-time options for performance prediction,” in Proc. 25th ACM Int. Syst. Softw. Product Line Conf. (SPLC), Leicester, U.K., Sep. 2021, vol. A, pp. 100– 111, doi: 10.1145/3461001.3471149. [6] R. Botvinik-Nezer et al., “Variability in the analysis of a single neuroimaging dataset by many teams,” Nature, vol. 582, no. 7810, pp. 84–88, Jun. 2020, doi: 10.1038/s41586-020-2314-9. [7] T. Glatard et al., “Reproducibility of neuroimaging analyses across operating systems,” Frontiers Neuroinformatics, vol. 9, Apr. 2015, Art. no. 12, doi: 10.3389/fninf.2015.00012. [8] E. H. Gronenschild, P. Habets, H. I. L. Jacobs, R.  Mengelers, N. Rozendaal, J. van Os, and M.  Marcelis, “The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements,” PLoS One, vol. 7, no. 6, Jun. 2012, A r t. no. e38234, doi: 10.1371/jour nal.pone. 0038234. 

SP

FROM THE GUEST EDITORS Bihan Wen , Saiprasad Ravishankar , Zhizhen Zhao , Raja Giryes , and Jong Chul Ye

Physics-Driven Machine Learning for Computational Imaging: Part 2

T

hanks to the tremendous interest from the research community, the focus of the March issue of the IEEE Signal Processing Magazine is on the second volume of the special issue on physics-driven machine learning for computational imaging, which brings together nine articles of the 19  accepted papers from the original 47 submissions.

Physics-driven machine learning has found many applications in computational imaging What makes computational imaging more exciting is its close relationship with real-world applications using imaging hardware. Unlike many other machine learning approaches to computer vision and image-processing problems that mainly deal with digitized images, machine learning approaches to computational imaging have their origins in real-world imaging hardware. Therefore, the physics-driven principle is tightly coupled to specific applications; hence, it is important to understand how the machine learning approaches can be integrated into the computational imaging pipeline using examples from realworld applications. Therefore, the review and tutorial articles in this March special issue aim to provide an overview of the real-world applications of recently proposed physicsdriven learning methods. Specifically, Digital Object Identifier 10.1109/MSP.2023.3236492 Date of current version: 17 February 2023

this edition aims to provide readers with more detailed information on how physics-based machine learning can be used to solve real-world imaging problems caused by electromagnetic (EM) waves, optics, and magnetic resonance imaging (MRI). The first two articles focus on imaging problems from EM waves, which are widely applied in sensing for security, biomedicine, geophysics, and various industries. Specifically, the article by Guo et al. [A1] provides informative background on EM imaging and basic formulations of the inverse problem. The authors then focus on three types of strategies combining physics and machine learning for linear and nonlinear imaging and discuss their advantages and limitations. The next article, by Su et al. [A2], focuses more on terahertz frequency-band imaging problems. As terahertz waves can partially penetrate through varieties of optically opaque objects and the rotational, vibrational, torsional frequencies of a great variety of molecules fall in the terahertz regime, terahertz imaging has been extensively studied for industrial inspection, security screening, chemical inspection, and nondestructive evaluation. In this article, the authors provide a detailed survey how learning-based approaches can be utilized to terahertz image restoration and reconstruction. As one might easily expect, a large part of the imaging problems in the real world arise from optics. The next three articles focus on the physics-driven machine learning approaches to solve optical imaging problems, such as IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

optical phase retrieval, hyperspectral unmixing, and even optical element design problems. Specifically, Pinilla et al. [A3] discuss hybrid approaches of model-driven networks or deep unfolding for the recovery of a complex signal from phaseless data acquired in the form of its diffraction patterns. Another important application of optical imaging is in remote sensing, which typically collect data using hyperspectral imaging sensors, allowing the identification of materials based on their unique spectral signatures that go beyond their visible properties. Although the classical approaches to hyperspectral imaging rely on explicit mixing models, these methods may not be accurate due to their limited modeling capabilities, especially when analyzing real scenes with unknown complex physical properties. In the article by Chen et al. [A4], the authors focus on combining the advantages of both physics-based models and data-driven methods to address the challenges in the hyperspectral unmixing problem. The next article, by Arguello et al. [A5], gives an overview of a different but very exciting way to use physics-driven machine learning for optical imaging. In contrast to the other two articles on optical imaging, which mainly focus on image reconstruction for specific optics, the authors examine a current research trend in the optics community that uses machine learning approaches for the design of optical encoding elements to achieve improved imaging physics. 13

The remaining four articles mainly focus on real-world medical imaging systems, such as X-ray and MRI. In fact, X-ray imaging could be regarded as another type of EM or optical imaging problem with X-ray ranges. However, the fundamental difference arises from tomography, where multiple projection images from different angles can be combined to reconstruct the internal structure of the object through inverse Radon transform. However, the basic assumption of the inverse Radon transform is that there are enough highquality projection images. Due to the potential for damaging healthy tissue, minimizing the radiation dose for X-ray computerized tomography has been extensively studied over the past two decades. The article by Xia et al. [A6] provides a survey of the physics-driven machine learning approaches to address the issue of high-quality tomographic reconstruction from low-quality projection data. Although MRI also relies on the microwave range, compared to all of the imaging problems mentioned above, MRI is unique in that it is based on Fourier coding, which allows for much higher resolution beyond the diffraction limit. However, due to the nature of Fourier coding, the acquisition time of MRI is significantly slower compared to other imaging approaches, and the main investigations into computational imaging in MR focus on accelerated acquisition. The article by Lam et al. [A7] deals with even more challenging problems of MRI, known as the MR spectroscopic imaging. The challenge here lies in the additional spectroscopic dimension in addition to spatial coding for MRI. Therefore, a naive way to use Fourier coding requires a significant amount of acquisition time, making it difficult to use in real medical applications. Therefore, the authors survey recent advance in this field by utilizing the physics-driven machine learning approaches. The article by Zhu et al. [A8] discusses physics-driven machine learning approaches for another timeconsuming MR acquisition model, quantitative MRI, which aims to obtain quantitative biophysical parameters 14

based on physical models derived from MR spin magnetization evolution. Although deep learning has already become a key element of MR acceleration, one of its bottlenecks is overfitting to insufficient training data. The article by Yang et al. [A9] reviews an emerging paradigm, imaging physics-based data synthesis, that can provide huge training data in biomedical MR without or with few real data. Although we have tried our best to provide a comprehensive overview of physics-driven machine learning approaches to computational imaging, we are aware that even the two volumes of the special issue are insufficient to cover many exciting developments in the field. While not sufficient, we hope that the overview of the theoretical foundations and practical applications can give readers a general overview of the field and encourage them to delve into this exciting area of research.

Acknowledgment We would like to thank the anonymous reviewers and the editor for their careful reading and very important comments to make this special issue successful.

Guest Editors Bihan Wen (bihan. [email protected]) received his B.Eng degree in electrical and electronic engineering from Nanyang Technological University, Singapore, in 2012 and his M.S and Ph.D. degrees in electrical and computer engineering from the University of Illinois at Urbana-Champaign in 2015 and 2018, respectively. He is a Nanyang assistant professor with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798. He serves as an associate editor of IEEE Transactions on Circuits and Systems for Video Technology. He was a recipient of the 2016 Yee Fellowship and the 2012 Professional Engineers Board Gold Medal of Singapore. He coauthored a paper that received the Best Paper Runner-Up Award at the IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

IEEE International Conference on Multimedia and Expo in 2020. His research interests include machine learning, computer vision, image and video processing, and computational imaging. He is a Member of IEEE. Saiprasad Ravishankar ([email protected]) received his B.Tech. degree in electrical engineering from the Indian Institute of Technology Madras, India, and his M.S. and Ph.D. degrees in electrical and computer engineering from the University of Illinois at UrbanaChampaign. He is an assistant professor in the Departments of Computational Mathematics, Sci­ence and Engineering, from 2015 to 2018, and Biomedi­cal Engineering at Michigan State University, East Lansing, MI 48824 USA. He did postdoctoral research in the Department of Electrical Engi­ neering and Computer Science at the University of Michigan from 2015 to 2018, and in the Theoretical Division at Los Alamos National Laboratory during 2018–2019. He organized special sessions or workshops on computational imaging at the 2016 IEEE Image, Video, and Multi­dimen­sional Signal Processing Workshop, 2017 IEEE International Workshop on Machine Learning for Signal Processing, 2018 IEEE International Symposium on Biomedical Imaging, and 2019 and 2021 International Conference on Computer Vision. He is a Senior Member of IEEE and an IEEE Computational Imaging Technical Committee member. Zhizhen Zhao (zhizhenz@illinois. edu) received her Ph.D. degree in physics from Princeton University and her B.A. and M.Sc. degrees in physics from Trinity College, Cambridge University. She is an associate professor and William L. Everitt faculty fellow in the Department of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign (UIUC), UrbanaChampaign, IL 61801 USA. Prior to

joining UIUC, she was a Courant instructor at the Courant Institute of Mathematical Sciences, New York University. She is a recipient of the Alfred P. Sloan Research Fellowship (2020–2022). Her research interests include applied and computational harmonic analysis, signal processing, and computational imaging. She is a Member of IEEE. Raja Giryes (raja@ tauex.tau.ac.il) received his Ph.D. degree from the Technion-Israel Institute of Technology. He is an associate professor in the School of Electrical Engineering at Tel Aviv University, Tel Aviv 69978, Israel. He received the European Association for Signal Processing Best Ph.D. Award, the European Research Council Starting Grant, the Maof prize for excellent young faculty (2016–2019), the VATAT scholarship for excellent postdoctoral fellows (2014–2015), the Intel Research and Excellence Award (2005, 2013), and the Excellence in Signal Processing Award from Texas Instruments (2008), and he was part of the Azrieli Fellows Program (2010–2013). He is an associate editor of IEEE Transactions on Image Processing and Elsevier’s Pattern Recognition and has organized workshops and tutorials on deep learning theory in various computer vision conferences. He is also a co-organizer of the Israel Computer Vision Day. He is a Senior Member of IEEE and has been a member of the Israeli Young Academy since 2022. Jong Chul Ye (jong. [email protected]) received his Ph.D. degree from Purdue University. He is a professor in the Graduate School of Artificial Intelligence and an adjunct professor in the Department of Bio/Brain Engineering and the Department of Mathematical Sciences at the Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Korea. Previously, he was a postdoctoral fellow at the University of Illinois at Urbana- Champaign, and a senior researcher at Philips Research and

GE Global Research. He is an associate editor of IEEE Transactions on Medical Ima­ging, a senior editor of IEEE Signal Processing Magazine, and an executive editor of Biological Imaging. He was a guest editor for several IEEE special issues. He is a Fellow of IEEE and was the chair of the IEEE Signal Processing Society Computational Imaging Technical Committee and an IEEE Engineering in Medicine and Biology Society Distinguished Lecturer. He was a general cochair for the 2020 IEEE International Symposium on Biomedical Imaging.

Appendix: Related Articles [A1] R. Guo, T. Huang, M. Li, H. Zhang, and Y. C. Eldar, “Physics-embedded machine learning for electromagnetic data imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 18–31, Mar. 2023, doi: 10.1109/MSP.2022.3198805. [A2] W.-T. Su, Y.-C. Hung, P.-J. Yu, C.-W. Lin, and S.-H. Yang, “Physics-guided terahertz computational imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 32–45, Mar. 2023, doi: 10.1109/MSP.2022.3198807. [A3] S. Pinilla, K. V. Mishra, I. Shevkunov, M. Soltanalian, V. Katkovnik, and K. Egiazarian,

[A4]

[A5]

[A6]

[A7]

[A8]

[A9]



“Unfolding-aided bootstrapped phase retrieval in optical imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 46–60, Mar. 2023, doi: 10.1109/MSP.2022.3214325. J. Chen, M. Zhao, X. Wang, C. Richard, and S. Rahardja, “Integration of physics-based and data-driven models for hyperspectral image unmixing,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 61–74, Mar. 2023, doi: 10.1109/MSP.2022.3208987. H. Arguello et al., “Deep optical coding design in computational imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 75–88, Mar. 2023, doi: 10.1109/MSP.2022.3200173. W. Xia, H. Shan, G. Wang, and Y. Zhang, “Physics-/model-based and data-driven methods for low-dose computed tomography,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 89–100, Mar. 2023, doi: 10.1109/MSP.2022.3204407. F. Lam, X. Peng, and Z.-P. Liang, “Highdimensional MR spatiospectral imaging by integrating physics-based modeling and datadriven machine learning,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 101–115, Mar. 2023, doi: 10.1109/MSP.2022.3203867. Y. Zhu, J. Cheng, Z.-X. Cui, Q. Zhu, L. Ying, and D. Liang, “Physics-driven deep learning methods for fast quantitative magnetic resonance imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 116–128, Mar. 2023, doi: 10.1109/MSP.2023.3236483. Q. Yang, Z. Wang, K. Guo, C. Cai, and X. Qu, “Physics-driven synthetic data learning for biomedical magnetic resonance,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 129–140, Mar. 2023, doi: 10.1109/MSP.2022.3183809.

SP

TAP. CONNECT. NETWORK. SHARE. Connect to IEEE–no matter where you are–with the IEEE App. Stay up-to-date with the latest news Schedule, manage, or join meetups virtually Get geo and interest-based recommendations Read and download your IEEE magazines Create a personalized experience Locate IEEE members by location, interests, and affiliations

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

15

2023.ieeeicip.org

International Conference on Image Processing OCTOBER 8 -11, 2023 The 30th IEEE International Conference on Image Processing (ICIP 2023) will be held in Kuala Lumpur, Malaysia, on October 8-11, 2023. ICIP is the world’s largest and most comprehensive technical conference focused on image and video processing and computer vision.

Topics of interest included but not limited to: • • • • • • • •

Sensing, Representation, Modeling, and Registration Motion Estimation, Registration, and Fusion Synthesis, Rendering, and Visualization Deep Learning for Images and Videos Computational Imaging Learning with Limited Labels Restoration and Enhancement Image & Video Interpretation and Understanding

Regular ICIP Paper Submission

Authors are invited to submit full-length papers (up to 4 pages for technical content including figures and references, and one optional 5th page containing only references). Submission instructions, templates for paper format, and the “no show” policy are available at the conference website. All accepted papers will be published in IEEE Xplore.

Open Journal of Signal Processing (OJSP) Paper Submission

For the first time at ICIP, authors are invited to submit their papers, aligned with the conference scope and intended for presentation at ICIP, to the IEEE Open Journal of Signal Processing (OJSP). These papers may be up to 8 pages + 1 page for references, and their review will be managed by the editorial board of OJSP, expedited to ensure that a decision is made prior to finalization of the ICIP technical program. Accepted papers will be published in OJSP and will also be scheduled for presentation at ICIP. Submission implies a commitment that an author will attend the conference to present the paper at ICIP if it is accepted. Acceptance of submissions to OJSP under this “expedited review for ICIP” program may close early if demand is unexpectedly heavy.

• • • • • • • •

Compression, Coding, and Transmission Detection, Recognition, Retrieval, and Classification Color, Multi-spectral, and Hyper-spectral Imaging Biometrics, Forensics, and Security Stereoscopic, Multi-view, and 3D Processing Biomedical and Biological Image Processing Image & Video Quality Models Emerging Applications and Systems

Journal Paper Presentation

Authors of papers published in all IEEE Signal Processing Society fully owned journals as well as in IEEE TCI, IEEE Signal Processing Letters, IEEE TIP, IEEE TMI, IEEE TM, IEEE TIFS, IEEE TSP, IEEE Journal on STSP, and IEEE SPM will be given the opportunity to present their works at ICIP 2023, subject to space availability and approval by the Technical Program Committee. Visit the conference website for more details.

Tutorials, Special Sessions, and Challenge Sessions Proposals

Tutorials will be held on October 8, 2023. Tutorial proposals must include a title, outline, description of the materials to be covered/distributed, and the contact information, biography and selected publications of the presenter(s). Special sessions and challenge sessions proposals must include a topical title, rationale, session outline, contact information, and a list of invited papers/participants. For detailed submission guidelines, please visit the conference website.

Open Preview Open Preview allows conference proceedings to be available in the IEEE Xplore Digital Library, free of charge, to all customers, 30 days prior to the conference start date, through the conference end date.

Author Rebuttal At the discretion of the Area Chairs, some authors may be asked for a rebuttal of the reviews received. Digital Object Identifier 10.1109/MSP.2023.3241815

SEE BACK FOR IMPORTANT DATES!

OCT. 8 -11

ORGANIZING COMMITTEE General Co-Chairs Norliza Mohd Noor (UTM) Gaurav Sharma (U. Rochester) Mohan Kankanhalli (NUS)

Technical Program Co-Chairs Eduardo AB da Silva (UFRJ) Stefan Winkler (NUS) Jing-Ming Guo (NTUST)

Finance Chair Mohammad Faizal Ahmad Fauzi (MMU) Aly Farag (Louisville U.)

Plenary Co-Chairs Shri Narayanan (USC) Syed Abdul Rahman Syed Abu Bakar (UTM)

Special Session Co-Chairs Chin Tat-Jun (Adelaide U.) Wong Lai Kuan (MMU)

Tutorials/Co-Located Chair Rajasvaran Logeswaran (City U.)

Publication Co-Chairs Chong-Wah Ngo (SMU) John See (HW)

Publicity Co-Chairs Syed Khaleel Ahmed (Consultant) Susanto Rahardja (SIT)

Industry/Exhibit Chair Liu JianQuan (NEC) Hezerul Abdul Karim (MMU)

Innovation Program/ Competition Co-Chairs Wong Kok Sheik (Monash) Jean Luc Dugelay (EUROCOM)

Local Arrangement Co-Chairs Vijanth Asirvadam (UTP) Hoo Wai Lam (UM)

Registration Co-Chairs Nor'aini Abdul Jalil (Wavesmiles) Haidawati Mohamad Nasir (UniKL)

Awards Co-Chairs Jocelyn Chanussot (Grenoble U.) Ma Kai Kuang (NTU)

Student Activity Co-Chairs (Post Doctoral Consortium) Kushsairy Abdul Kadir (UniKL) Fabrice Meriaudeu (Bourgogne U.)

Social Media Chair Mohd Norzali Haji Mohd (UTHM) Loh Yuen Peng (MMU)

Keynote Chair Cheng Wen-Huang (NYCU) Ba- Ngu Vo (Curtin U.)

Supported by:

IMPORTANT DATES Challenge Proposal Deadline: November 23, 2022 Challenge Proposal Acceptance Notification: December 14, 2022 Special Session Proposal Deadline: December 7, 2022 Special Session Proposal Acceptance Notification: January 4, 2023 Tutorial Proposal Deadline: January 11, 2023 Tutorial Proposal Acceptance Notification: February 1, 2023 Regular & Special Session Paper Submission Deadline: February 15, 2023 Challenge Paper Submission Deadline: March 1, 2023 Journal Presentation Request Deadline: May 31, 2023 Paper Acceptance Notification: June 21, 2023 Journal Presentation Acceptance Notification: June 21, 2023 Final Paper Submission Deadline: July 5, 2023 Author Registration Deadline: July 12, 2023

Connect with us!

PHYSICS-DRIVEN MACHINE LEARNING FOR COMPUTATIONAL IMAGING

Rui Guo   , Tianyao Huang   , Maokun Li   , Haiyang Zhang  , and Yonina C. Eldar 

Physics-Embedded Machine Learning for Electromagnetic Data Imaging Examining three types of data-driven imaging methods

E

lectromagnetic (EM) imaging is widely applied in sensing for security, biomedicine, geophysics, and various industries. It is an ill-posed inverse problem whose solution is usually computationally expensive. Machine learning (ML) techniques and especially deep learning (DL) show potential in fast and accurate imaging. However, the high performance of purely datadriven approaches relies on constructing a training set that is statistically consistent with practical scenarios, which is often not possible in EM-imaging tasks. Consequently, generalizability becomes a major concern. On the other hand, physical principles underlie EM phenomena and provide baselines for current imaging techniques. To benefit from prior knowledge in big data and the theoretical constraint of physical laws, physicsembedded ML methods for EM imaging have become the focus of a large body of recent work. This article surveys various schemes to incorporate physics in learning-based EM imaging. We first introduce background on EM imaging and basic formulations of the inverse problem. We then focus on three types of strategies combining physics and ML for linear and nonlinear imaging and discuss their advantages and limitations. Finally, we conclude with open challenges and possible ways forward in this fast-developing field. Our aim is to facilitate the study of intelligent EM-imaging methods that will be efficient, interpretable, and controllable.

©SHUTTERSTOCK.COM/PAPAPIG

Digital Object Identifier 10.1109/MSP.2022.3198805 Date of current version: 17 February 2023

18

Introduction EM fields and waves have long been used as a sensing method. This is because the EM field can penetrate various media, interact with materials, and alter its distribution in both space and time. Hence, electric and magnetic properties of materials, such as permittivity, permeability, and conductivity, can be inferred from field samples. EM imaging refers to reconstructing the value distribution of electric or magnetic parameters from measured EM fields, through which a better understanding of the domain of investigation (DoI) can be obtained. EMimaging techniques have been widely applied in security, biomedicine, geophysics, and various industries. In security, for example, EM imaging using radars can help locate targets that

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

1053-5888/23©2023IEEE

record the field at specific locations, i.e., the field samples are sparse in space. The attenuation of EM fields caused by diffraction and absorption of media, as well as a noisy environment, further increases the ill-posedness. ■■ Solving EM-imaging problems often requires accurate modeling of EM wave propagation in the DoI. This process is called forward modeling and is implemented by numerical algorithms such as the finite-element method. However, it is computationally intensive, especially for large DoIs. Recent advances in big data storage, massive parallelization, and optimization algorithms have facilitated the development of ML and its applications in EM imaging [2], [3], [4], [5]. ML is attractive for overcoming the aforementioned limitations due to the following aspects. First, the time-consuming operations of modeling and inversion can be surrogated by data-driven models to make imaging faster. Second, prior knowledge that is difficult to describe with rigorous forms can be recorded after the learning process, which helps improve imaging accuracy. Finally, DL software frameworks provide user-friendly interfaces to fully exploit computing power without low-level programming on heterogeneous platforms, which largely reduces the complexity of algorithm implementation for high-performance imaging. Training a surrogate model for data-image mappings such as deep neural networks (DNNs) has shown promising results [3]. However, success relies on constructing a training dataset that is statistically consistent with practical scenarios. Due to multiple scattering effects, simply establishing the mapping from EM data to electric properties by black-box regression may lead to implausible predictions, even when the measured data are not highly out of distribution. On the other hand, physical laws provide baselines for EM imaging. The relationship between EM fields and electric properties is inherent in Maxwell’s equations. Recent trends show that a hybrid of physics- and data-driven methods can analyze and predict data more effectively [5]. Such techniques can be grouped into learning-assisted physics-driven techniques and physics-embedded ML approaches. The  former category solves the inverse problem in physicsbased frameworks, where learning approaches are applied to

are invisible to optical imaging. In biomedicine, microwave imaging can detect anomalies in the permittivity distribution caused, e.g., by a cerebral hemorrhage. As a final example, images of conductivity distribution reconstructed from lowfrequency EM fields may reveal deep structures in the earth. A theoretical model of EM imaging is illustrated in Figure 1, where EM sensors, i.e., antennas, are deployed around the DoI. When an external source illuminates the DoI, sensors record the EM field. Given the transmitting waveform as well as locations of the transmitting antennas, the EM field propagates according to Maxwell’s equations [1]. In the frequency domain, EM propagation can be described by the following partial differential equation (PDE):

d # d # E (r) - ~ 2 ne (r) E (r) = i~nJ (r) (1)

where E is the vector electric field, r is the spatial position, n is permeability, e is complex permittivity, J is the electric current source, ~ is the angular frequency, and d # is the curl operator. Complex permittivity is expressed as e = e R + iv/~, where its real part e R is permittivity and its imaginary part is related to conductivity v. Equation (1) can describe both wave physics (e R & v/~) and diffusion physics (v/~ & e R), depending on the settings of investigation. Here permeability is assumed to be constant, which is reasonable in most imaging scenarios. In EM imaging, we usually have information about the sources; therefore, ~ and J are both known. Once we measure the electric field at the receiver locations, complex permittivity can be recovered by solving (1). This process defines EM inverse problems in which EM parameters are solved given measured electric fields. The EM inverse problem is nonlinear and often ill-posed due to the following several challenges: ■■ The nonlinearity comes from complex interactions between the measured EM field and the material parameters. As we can see from (1), the product of E and e results in a nonlinear relationship where the nonlinearity increases with e. ■■ The ill-posedness arises from multiple scatterings, insufficient measurements, and noise corruption. Due to multiple scatterings of EM waves, a slight variation of targets may change the EM field substantially. In most cases, we only

Transmitting Antenna

Receiving Antenna

Permittivity Imaging y

Measurement

Index of Receivers

Electric Field 1 10 20 30 40 50 60

1 6 12 18 Index of Transmitters

x

FIGURE 1. The EM-imaging setup. EM imaging converts measured data to the spatial distribution of electric parameters in the DoI. IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

19

augment performance, such as generating better initial guesses [6], improving the bandwidth of measured data [7], or encoding prior knowledge [8]. The latter category performs imaging mainly in data-driven manners, where algorithms are designed according to physical laws, such as tailoring inputs and labels [9], [10], [11], [12], loss functions [13], [14], [15], and neural network structures [16], [17], [18]. Although frameworks of physics-based techniques have been well studied, the learning methods for EM imaging vary widely. This article aims to review recent frontiers in physicsembedded ML for EM-imaging techniques and shed light on designing efficient and interpretable ML-based imaging algorithms. Existing approaches include modeling Maxwell’s equations into the learning process and combining trainable parameters with full-wave EM solvers or differential/integral operators [13], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25]. These ML models not only describe physical principles but also record the prior knowledge gained from training data. Compared with purely data-driven models, physics-embedded approaches possess higher generalizability and can learn effectively from fewer training data [26]. In addition, as there is no need to train the physics part, both the memory and computation complexity of DNNs can be reduced. We begin in the “Formulations and Challenges of EM Imaging” section by introducing basic formulations of the EM imaging problem and stating some of the challenges with conventional methods. We then categorize existing physicsembedded ML approaches into three kinds: learning after physics processing, learning with physics loss, and learning with physics models in the “Learning After Physics Processing,” “Learning With Physics Loss,” and “Learning With Physics Models” sections, respectively. The “Challenges and Opportunities” section discusses open challenges and opportunities in this fast-developing field. We draw conclusions in the “Conclusions and Outlooks” section.

Formulations and challenges of EM imaging EM imaging is an inverse problem that calculates electric parameters of the DoI from measured EM fields. This process incorporates EM modeling, which simulates “measured” data based on numerical models. It can be described as minimizing the “misfit” between the observed and simulated data [27]

L (e) = d obs - F (e)

2

+ mz r (e) (2)

where d obs is the field observed by receivers, e is complex permittivity, F (e) represents the EM-modeling function, z r is the regularization term, and m is a regularization factor. Note that “complex permittivity” is usually simplified to “conductivity” in low-frequency EM methods. This article uses “complex permittivity” to represent the unknown in both high- or low-frequency methods. EM modeling, F (e), computes the EM field in space given the permittivity distribution in the DoI and the information on the sources. It is usually achieved by numerical means, 20

e.g., the finite-element and finite-difference techniques and the methods of moments [28]. These procedures partition the DoI into thousands or millions of subdomains and convert the wave equation into a matrix equation. EM fields in space are obtained by solving the matrix equation that involves thousands or millions of unknowns. The solution process can take minutes or hours. This computationally expensive process is usually called full-wave simulation. To accelerate the modeling process, one can make approximations so that the EM field is linear in the electric parameters, such as the Born or Rytov approximations [1]. In this linearized process, the EM field is computed by simple matrix operations, such as matrix-vector multiplications or the Fourier transform. The regularization, z r (e), is used to incorporate prior knowledge into the imaging process, and it varies in different tasks. For instance, in geophysical or biomedical imaging, to emphasize the sharpness or smoothness of material boundaries, , 1 and , 2 norms of the spatial gradient of e are usually adopted [29], [30], [31], [32], [33]. In radar imaging, the sparsity of the observed scene is often exploited to improve imaging quality by incorporating sparsity regularization, such as the , 1 norm given by z r (e) = e 1 [34], [35], [36]. Equation (2) is usually minimized by iterative gradient descent methods, but some challenges still exist. For instance, each iteration requires computing the forward problem and its Fréchet derivative. Many forward problems are computationally intensive, and computing the Fréchet derivative of F (e) with respect to e, i.e., S = 2F (e) /2e, needs to call the forward solver F (e) many times, which exacerbates computational burden. Some fast algorithms that compute approximations of Fréchet derivatives have been proposed [27], [37], [38], but the solution process still needs to be accelerated. Furthermore, when F (e) is rigorously solved from Maxwell’s equations, the objective function is nonconvex and has numerous local minima due to nonlinearity between EM responses and permittivity. Finally, gradient descent methods lack flexibility in exploiting the prior knowledge that is not described by simple regularization. On the other hand, stochastic inversion schemes have been developed to cope with the necessity of reliable uncertainty estimation and to have a natural way to inform imaging with realistic and complex prior information [39], [40], [41]; however, stochastic sampling is often computationally intensive. Physics-embedded ML models provide potential solutions to the challenges mentioned in this section. In the following, we present three types of physics-embedded models for EM imaging, as depicted in Figure 2. The first class processes EM data using conventional physical methods and ML models sequentially [9], [10], [11], [42], [44]. The second type optimizes network parameters with physics constraints, for example, solving forward problems in the training loss function [13], [15], [19]. The third category unrolls the physical methods with neural networks [16], [17], [18], [20], [21], [22], [23], [24], [25]. These three approaches are discussed in detail in the following three sections.

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

Learning after physics processing

scatterers in a through-wall imaging test within 1 s. Structural similarity improves by more than 50% compared to convenThe learning after physics processing approach consists of tional algorithms. two sequential steps: first, a roughly estimated image is reIt should be noted that the more the input is processed by covered using classical qualitative or quantitative methods; physics, the better the generalizability will be. For instance, the second, the rudimentary image is polished using a DNN Wei and Chen [10] compare performances with various nettrained with the ground truth as labels. In this approach, the work inputs, including raw, scattered data; permittivity from tasks of DNNs become image processing, such as eliminating backprojection; and permittivity from the dominant current artifacts and improving resolution. Next we introduce several scheme (DCS). The network behaves the poorest when directclassical works that utilize DNNs to enhance image quality in ly inputting raw data, and best when inputting the image from this direction. the DCS. This is because preprocessing in the DCS involves In conventional EM imaging, permittivity is iteratively more physics and thus reduces the network’s burden. A simirefined by updating in the descent directions of the objective lar conclusion is drawn in [45], where the input and output of function. This motivates employing multiple convolutional neural networks are preprocessed with the wave-propagation neural network (CNN) modules to progressively improve operator, i.e., Green’s function, to a deeper degree, achieving image resolution, starting from some rudimentary images. better performance on accuracy and robustness against noise These images can come from conventional imaging methods, than the DCS [10] when recovering high-permittivity targets. including linear backprojection [9], subspace optimization The sequential workflow provides great flexibility in borrow[10], one-step Gauss–Newton [11], and contrast source-invering well-developed DL techniques in image processing, and most sion techniques [42]. Finally, the CNN can output superof the mentioned works can achieve real-time imaging. Recent resolution images close to the ground truth. To construct the works have extended this approach to uncertainty quantification training dataset, many researchers convert the handwritten of imaging results [46]. However, although a DNN can generate digits dataset to permittivity models, then perform full-wave a plausible image, the recovered permittivity values may signifisimulations to obtain the scattered electric data. To train the cantly differ from true ones. This is because the DNN is trained DNN, the rudimentary image is taken as the input, while without the supervision of the EM field. In the following section, the corresponding true-permittivity image is the label. After we introduce another group of methods that takes into account training with handwritten letters, the DNN predicts targets fitness between the computed and measured data. with more complex shapes and permittivity. Advanced DNN architectures may improve the performance of image enhancement. The U-Net [5], [74], one Learning After Learning With Learning With of the most widely used architectures, is Physics Processing Physics Loss Physics Models built on the encoder-decoder architecture and has skip connections bringing Measurement Measurement Measurement encoded features to the decoder. This ensures feature similarity between the Physics Model input and output and is especially suitable for superresolution. For example, the authors in [10] and [42] use U-Nets to Raw Image achieve superresolution for 2D microwave imaging. In [44], the 3D inverse scattering problem is solved by a 3D U-Net, where the input is the preliminary 3D Physics-Guided ML With ML Architecture model recovered by Born approximation Physics-Guided Loss Function inversion and the Monte Carlo method. ML After Physics Another architecture for superresoProcessing lution is a generative adversarial network (GAN). In [11] and [12], a GAN Image Image Image with cascaded, object-attentional superresolution blocks is applied to imaging with an inhomogeneous background. Trainable Parameters Physics Operators The authors use a GAN with an attention scheme to improve the resolution by highlighting scatterers and inhibiting FIGURE 2. The three ways of incorporating physics into the ML model. (a) Learning after physics prothe artifacts. In [11], after training with cessing: the physics model is employed to initialize the input of ML models. (b) Learning with physics 6,000 handwritten digit scatterers, the loss: physics knowledge is incorporated into the loss functions. (c) Learning with physics models: GAN reconstructs U-shape plexiglass physics knowledge is used to guide design of the ML architecture. IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

21

Learning with physics loss This section presents several approaches that impose additional physical constraints on the loss function when t­ raining the ­network weights, different from conventional ML models, which use only the difference between predicted and labeled images as a loss function. The advantage of additional constraints in loss is demonstrated in “Incorporating Forward Modeling in Loss: A Mathematical Example.” In the following, we consider different types of physics losses: rigorous measurement, learned measurement, and PDE-constrained.

Training with a rigorous measurement loss Consider the inverse problem solved by a DNN with the measured data d as the input and the permittivity e as the output. When the EM data at the receivers can be numerically computed as in many applications, one can embed the data fitness, which involves physical rules, in the training loss function [13]. Let e T and d T denote the labeled permittivity and EM data, respectively, for training. Purely data-driven imaging 2 uses permittivity loss L e = e - e T for training. The phys-

ics-embedded one further incorporates the measurement (data) loss L d, given by

L = aL e + bL d = a e - e T

2

+ b F ^ e h - d T (3) 2

where a and b are weighting coefficients. If a = 0, the DNN training can be regarded as unsupervised learning. In the geosteering EM data inversion [13], this scheme achieves a twoorders-of-magnitude-lower data misfit compared to training with permittivity loss only (b = 0). Training such a DNN requires backpropagating the gradients of the data misfit, where the Fréchet derivative 2F/2e needs to be computed outside the DL framework. The methods for estimating the derivative have been addressed in traditional deterministic inversion, such as the finite-difference or adjointstate technique [47].

Training with a learned measurement loss Computing the Fréchet derivative 2F/2e is time consuming. An idea to accelerate it is to surrogate the numerical forward

Incorporating Forward Modeling in Loss: A Mathematical Example Optimizing deep neural network (DNN) parameters with the constraint of a forward problem can alleviate the ambiguity caused by nonuniqueness of the inverse problem. We demonstrate it using a toy problem [19], where the forward process has analytical solutions:

m|= F ( p) = p 2. (S1)

The inversion has two branches of solutions, p = + m and p = - m , see the black lines in Figure S1. To solve the inverse problem with a DNN, the training dataset is

constructed such that for each sample ^ m, m h , there is another one ^ m, - m h in it. The point that simultaneously minimizes the distance between the two solutions is zero. When training is supervised by labels p ^ ! m h , the predictions are zeros [see Figure S1(a)], which are fake answers caused by the nonuniqueness. On the other hand, when training is supervised by labels p 2 , the correct branch can be predicted by controlling the signs of solutions [see Figure S1(b)]. In EM imaging, the forward problems are described by Maxwell’s equations.

Training Supervised by ±√ m

Training Supervised by m 33

33

Predicted

0

Predicted

I (m)

I (m)

Real

0

Real –33

–33 0

500 m (a)

0

1,000

500 m (b)

1,000

FIGURE S1. Incorporating forward modeling into training to reduce nonuniqueness of the inverse problem [19]. (a) When training is supervised by p ( ! m ), the predictions are zeros and (b) when training is supervised by labels p 2, the correct branch can be predicted by controlling the signs of solutions.

22

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

solver F (·) with a DNN H F (·) [19]. The training contains two stages: 1) training the forward solver H )F and 2) training the inverse operator H )I , given by 2

H )F = arg min H F (e T ) - d T , HF



2

H )I = arg min H )F (H I (d T )) - d T .



HI

(4)

Both stages take the measurement misfit as the loss function, which involves physical rules. This scheme successfully solves the logging-whiledrilling inverse problem for borehole imaging [19], (see Figure 3) where the purely data-driven approach did not achieve satisfactory reconstructions due to the severe nonlinearity and ill-posedness in logging-while-drilling inversion [48].

A PINN is applied to electrical impedance tomography in [15]. Electrical impedance tomography measures the voltages on a body surface after electric currents are injected. The imaging recovers conductivity distribution inside the body. It usually contains multiple transmitting and receiving sensors. In “A Physics-Informed Neural Network for the Inverse Problem,” we show that a PINN contains one forward network H F and one inverse network H I for one transmitting source. When J sources illuminate the domain, the PINN will contain J + 1 networks, corresponding to J forward networks that output voltages generated by different sources and one inverse network that outputs conductivity. Furthermore, the loss function should be modified to L = R iJ L i, where L i represents the loss function for the ith source. Simultaneously training all networks can satisfy both PDEs and boundary conditions (measurements).

Training with a PDE-constrained loss

Resistivity (Ω-m) 5 10 20 1.0e+02

1.0e+00 Depth

The PDE-constrained loss inserts PDEs into the loss function. The representative work is the physics-informed neural network (PINN) [14], [49], which is designed for both forward and inverse problems. It is a meshfree method and can seamlessly fuse knowledge from observations and physics. We present an example of a PINN for the inverse problem in “A Physics-Informed Neural Network for the Inverse Problem.”

Horizontal Offset

FIGURE 3. Underground resistivity predicted from logging-while-drilling EM data [19]. The gray curve represents drilling trajectory, along which EM fields are transmitted and collected by a logging instrument. Resistivity around the trajectory is recovered by a DNN trained with a learned measurement loss.

A Physics-Informed Neural Network for the Inverse Problem Consider the 1D time-domain electromagnetic wave equation

2 2 E (x, t) 2 2 E (x, t) - ne (x) = 0 (S2) 2 2x 2t 2

where E is the electric field, e is permittivity, n is permeability, and t and x are the time and spatial coordinate, respectively. Together with some boundary conditions, the equation can be analytically or numerically solved to yield E (forward problem) or e (inverse problem) given t and x. Take the inverse problem with one-source multiple receivers as an example. A physics-informed neural network (PINN) specifies two separate deep neural networks (DNNs), namely, H F and H I . The input of H F is x and t and its output is the electric field Eu , denoted by Eu = H F (x, t). Similarly, the input of H I is x and its output is permittivity eu, denoted by eu = H I (x). The two separate DNNs are simultaneously trained with a shared loss function L, which includes a supervised measurement loss of E regarding initial and boundary conditions

1 L data = N | ^ Eu ^ x i, t i h - E T ^ x i, t i hh2 (S3) data i =1 N data



and an unsupervised loss of partial differential equation constructed according to (S2) N 2 2 Eu ^ x j, t j h 2 2 Eu ^ x j, t j h 2 1 m (S4) L PDE = N | c - neu (x j) 2 PDE 2x 2t 2 j=1 PDE

given by L = a data L data + a PDE L PDE . Here ^ x i, t i h and ^ x j, t j h are sampled at the initial/boundary position and in the domain of investigation (DoI), respectively. In addition, E T is the labeled measurement, N data is the number of labeled samples, N PDE is the number of unlabeled samples in the DoI, and a · are weights. The partial differentiations are achieved by the automatic differentiation in the deep learning framework. After training, one can use H I to predict permittivity at arbitrary location x. Therefore, the PINN is mesh free.

IEEE SIGNAL PROCESSING MAGAZINE

|

March 2023

|

23

Following the use of unrolling in other domains [17], [50], [51], unrolling has also been used in EM-imaging models, yielding physics-embedded neural networks. We group this type into three subtypes: unrolling the measurement-to-image (inverse), unrolling the image-to-measurement (forward), and simultaneously unrolling both mappings.

max (·, 0), and i is the threshold. The ISTA shows high accuracy but requires thousands of iterations for convergence. To accelerate the solution process, the learned ISTA (LISTA) with only several neural network layers is proposed in [53], where each layer unfolds ISTA iterations. Particularly, the LISTA treats m/L, (1/L) U H and (I - (1/L) U H U) in (5) as variables to learn from training data with a backprojection algorithm, disregarding their physics structures. The numerical results in [53] show that the LISTA can achieve virtually the same accuracy as the ISTA using nearly two-order fewer iterations and does not require knowledge of U. Nevertheless, a challenge in the LISTA is that there are many variables to learn, requiring careful tuning of hyperparameters to avoid overfitting and gradient vanishing. Embedding physics models into the neural networks reduces the number of variables while maintaining fast convergence rate [23], [24]. For example, the mutual inhibition matrix I - (1/L) U H U has a Toeplitz or a doubly block Toeplitz structure due to the nature of radar-forward models. The degrees of freedom with such a Toeplitz structure are reduced to O(N) from O (N 2), the counterpart without this structure. By incorporating such a structure, the proposed method in [23] significantly reduces the dimension of neural networks, thereby reducing the amount of training data, memory requirements, and computational cost, while maintaining comparable imaging quality as the LISTA. A similar approach is also adopted in [24], which explores the coupling structure between different blocks in the radarforward model U.

Unrolling measurement-to-image mapping

Nonlinear problem

The smoothness of conductivity and known conductivity on the boundary are represented as regularizations in the loss function of a PINN to stabilize the inverse process [15]. In numerical simulations, the authors set J = 8, N data = 10, 000, and N PDE = 8, 000 and achieve better results than two conventional methods. However, one should note that we seldom have so many measurements in reality, so its performance on experimental data imaging needs to be further investigated.

Discussions When inverting limited-aperture EM data, insufficient measurements may lead to the instability of training a PINN. In this case, it would be better to use the first two approaches that explicitly define the measurement loss at the receivers. A PINN outperforms the two approaches when simulating the EM response is prohibitive, for example, due to the high computational cost or complex EM environment. Finally, when recovering diverse targets, the former two approaches can make predictions without retraining the neural network, while a PINN needs to be trained for each target.

Learning with physics models

Linear problem We demonstrate the unrolling of linear inverse problems through radar imaging. Here, the electric parameters of interest are intensities of scatterers in the DoI, denoted by e with a slight abuse of notation. Then, linear approximation is usually applied in the forward model for high computational efficiency, yielding F (e) = Ue, where U is a matrix determined by the radar waveform and the geometry of the DoI. Conventional radar imaging can be formulated as a compressed sensing problem: min e < d obs - Ue