Cnn: A Paradigm For Complexity: 31 (World Scientific Series on Nonlinear Science Series A) [Illustrated] 981023483X, 9789810234836

Citation preview

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

CNN: fl PRRRDIGN FOR COMPLEXITY

WORLD SCIENTIFIC SERIES ON NONLINEAR SCIENCE Editor: Leon 0. Chua University of California, Berkeley

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Series A.

MONOGRAPHS AND TREATISES

Published Titles Volume 15:

One-Dimensional Cellular Automata B. Voorhees

Volume 16:

Turbulence, Strange Attractors and Chaos D. Ruelle

Volume 17:

The Analysis of Complex Nonlinear Mechanical Systems: A Computer Algebra Assisted Approach M. Lesser

Volume 19:

Continuum Mechanics via Problems and Exercises Edited by M. E. EglitandD. H. Hodges

Volume 20:

Chaotic Dynamics C. Mira, L Gardini, A. Barugola andJ.-C. Cathala

Volume 21:

Hopf Bifurcation Analysis: A Frequency Domain Approach G. Chen andJ. L Moiola

Volume 22:

Chaos and Complexity in Nonlinear Electronic Circuits M. J. Ogorzalek

Volume 23:

Nonlinear Dynamics in Particle Accelerators ft Dilao and ft Alves-Pires Chaotic Dynamics in Hamiltonian Systems H. Dankowicz

Volume 25: Volume 30: Volume 31:

Quasi-Conservative Systems: Cycles, Resonances and Chaos A. D. Morozov CNN: A Paradigm for Complexity L. O. Chua

Forthcoming Titles Volume 4:

Methods of Qualitative Theory in Nonlinear Dynamics (Part I) L Shilnikov, A. Shilnikov, D. Turaev and L. O. Chua

Volume 18:

Wave Propagation in Hydrodynamic Flows A. L Fabrikant and Y. A. Stepanyants

Volume 27:

Thermomechanics of Nonlinear Irreversible Behaviours G. A. Maugin

Volume 32:

From Order to Chaos II L P. Kadanoff

|.|WORU)SCIENTinCSeRIESON|-%

C«,I««» A

NONLINEAR S C I E N C E ^

series A voi.31

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Series Editor: Leon 0. Chua

\I«I

01

CHN: I PRRRDIGM FOR COMPLEXITY

Leon 0. Chua University of California, Berkeley

V f e World Scientific wb

Singapore • NewJersey JerseyL• London • Hong Kong Singapore'New

Published by World Scientific Publishing Co. Pte. Ltd. P O Box 128, FarrcrRoad, Singapore 912805 USA office: Suite IB, 1060 Main Street, River Edge, NJ 07661

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

UK office: 57 Shclton Street, Covent Garden, London WC2H 9HE

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

CNN: A PARADIGM FOR COMPLEXITY Copyright © 1998 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

ISBN 981-02-3483-X

Printed in Singapore by U t o P r i n t

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

This treatise is dedicated

to CNN students of the 21st century

who will have been enlightened by

the LOCAL ACTIVITY DOGMA

and the EDGE OF CHAOS.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

This page is intentionally left blank

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

PREFACE

This treatise presents a new paradigm of EMERGENCE and COMPLEX­ ITY, with applications drawn from numerous disciplines, including artificial life, biology, chemistry, computation, physics, image processing, information science, etc. CNN is an acronym for Cellular Neural Networks when used in the con­ text of brain science, or Cellular Nonlinear Networks, when used in the context of emergence and complexity. A CNN is modeled by cells and inter­ actions: cells are defined as dynamical systems and interactions are defined via coupling laws. The CNN paradigm is a universal Turing machine and includes cellular automata and lattice dynamical systems as special cases. While the CNN paradigm is an example of REDUCTIONISM par excel­ lence, the true origin of emergence and complexity is traced to a much deeper new concept called local activity. The numerous complex phenomena unified under this mathematically precise principle include self organization, dissipative structures, synergetics, order from disorder, far-from-thermodynamic equilibrium, collective behaviors, edge of chaos, etc. The central theme of this treatise asserts that the somewhat fuzzy notions of "emergence" and "complexity", as well as their various metamorphosis, such as those cited above, can all be rigorously explained by a precise scien­ tific paradigm abstracted mathematically from the principle of conservation of energy; namely, a CNN operating near the edge of chaos, where the cells are not only locally active, but also linearly asymptotically stable. In particular, constructive and explicit mathematical inequalities are given for identifying the region in the CNN parameter space where complex phenom­ ena may emerge, as well as for localizing it further into a relatively small parameter domain called the edge of chaos where the potential for emergence is maximized.* The content of this monograph is based on a lecture presented at a work­ shop entitled "Visions of nonlinear science in the 21st century," held in Seville, Spain on 26 June 1996, the proceedings of which is published as a 'For an in-depth application of this provocative new theory, see R. Dogaru and L. O. Chua, "Edge of chaos and local activity domain of FitzHugh-Nagumo Equation," International Journal of Bifurcation and Chaos, Vol. 8, No. 2, 1998.

vii

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

viii

Preface

theme issue of the International Journal of Bifurcation and Chaos, Vol. 7, No. 10, 1997. As already pointed out in the opening remark of t h a t lecture, the key idea of connecting "complexity" with the hitherto unrelated concept of "local activity" was born the day before the lecture on a sandy beach of Costa del Sol while the author was brainstorming for what would turn out to be a historic lecture. Many friends, colleagues, former students, and research scholars from the Berkeley N O n l i n e a r ELectronics Laboratory (NOEL) have contributed to the emergence of this complex monograph. I would like to take this oppor­ tunity to thank in alphabetical order, L. Cooper, K. Crouse, R. Dogaru, M. Hasler, A. S. Huang, J. Huertas, T. Kozek, R. N. Madan, G. Moschytz, J. Neirynck, J. Nossek, A. Perez-Munuzuri, T. Roska, G. Setti, B. Shi, K. Slot, I. Szatmari, P. Thiran, R. Tetzlaff, F . Werblin, C. W. Wu, T. Yang and A. Zarandy. I alone, however, am responsible for whatever oversights and shortcom­ ings of any aspects of this book. Leon Chua Berkeley, California March 1, 1998

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

CONTENTS

Preface Introduction 1. W h a t is a CNN? 2. Part I: Standard CNNs 2.1. Standard CNNs are uniquely specified by CNN genes . . . 2.2. Oscillations and chaos from standard CNNs 2.3. Complete stability criteria for standard CNNs 2.4. Bistable criterion 2.5. Coding the CNN gene 2.5.1. Edge detection CNN 2.5.2. Corner detection CNN 2.6. A gallery of basic CNN genes 2.7. Does there exist a CNN gene for solving Minsky's global connectivity problem? 2.8. Decoding the CNN gene 2.8.1. Examples of input output CNN operators 2.8.2. Uncoupled CNN genes 2.8.3. Boolean CNN genes and t r u t h tables 2.9. W h a t task can an uncoupled Boolean CNN gene Perform? 2.10. Bifurcation of CNN genes 2.11. The game-of-life CNN gene 2.12. The CNN universal machine 2.13. Generalized cellular automata 2.14. A glimpse at some real-world CNN applications 3. Part II: Autonomous CNNs 3.1. Pattern formation in standard CNNs 3.1.1. Characterization of stable equilibria 3.1.2. T h e dynamics of pattern formation 3.1.3. CNN pattern formation in biology and physics . . . 3.2. Pattern formation in reaction-diffusion CNNs 3.3. Nonlinear waves in reaction-diffusion CNNs 3.4. Simulating nonlinear P D E s via autonomous CNNs . . . .

ix

vii 1 5 16 16 19 22 27 32 33 39 43 92 109 110 Ill 116 131 136 143 155 168 189 206 208 208 211 217 226 235 253

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

x

Contents

4. Part III: Local Activity: The Genesis of Complexity 4.1. Transistors and local activity: What do they have in common? 4.2. Nonlinear circuit models for reaction-diffusion CNNs 4.3. What is local activity? 4.3.1. Cell equilibrium points 4.3.2. Local state equations and local power flows . 4.3.3. Local activity in reaction-diffusion CNN cells . 4.4. Testing for local activity 4.4.1. Testing one-port CNN cells for local activity . 4.4.2. Testing two-port CNN cells for local activity . 4.5. Why is local activity necessary for pattern formation? 4.6. How to choose locally-active CNN parameters? 4.7. Local activity and stability are different concepts 4.8. The local activity dogma Index

266 267 . . . 268 276 278 . . . 281 . . . 286 288 . . . 288 . . . 293 . . . 298 300 300 302 311

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

INTRODUCTION CNN is an acronym for either Cellular Neural Network when used in the context of brain science, or Cellular Nonlinear Network when used in the context of coupled dynamical systems. A CNN is defined by two mathemat­ ical constructs: 1. A spatially discrete collection of continuous nonlinear dynamical systems called cells, where information can be encrypted into each cell via three independent variables called input, threshold, and initial state. 2. A coupling law relating one or more relevant variables of each cell dj to all neighbor cells Cki located within a prescribed sphere of influence Sij(r) of radius r, centered at dj. In the special case where the CNN consists of a homogeneous array, and where its cells have no inputs, no thresholds, and no outputs, and where the sphere of influence extends only to the nearest neighbors (i.e. r = 1), the CNN reduces to the familiar concept of a nonlinear lattice. Part I of this three-part exposition (reprinted from a theme issue on "Vi­ sions of Nonlinear Science in the 21st Century" of the International Journal of Bifurcation and Chaos, volume 7, numbers 9 & 10, 1997) is devoted to the standard CNN equation Xjj

—

27

X]

>

a

klVkl +

^2

bkiUki + Z*3

kl£Sij(r) X

Vij = f( ij)

;

= ^(\^j

+ M-~ \Xij ~ 1|)

* = 1, 2,...,M,j

= 1 , 2 , . . . , TV

where Xij, yij, uzj and zij are scalars called state, output, input, and threshold of cell Cij: aw and b^i are scalars called synaptic weights, and Sij(r) is the sphere of influence of radius r. In the special case where r = 1, a standard CNN is uniquely defined by a string of "19" real numbers (a uniform threshold Zki = z, nine feedback synaptic weights aki, and nine control synaptic weights b^) called a CAW gene because it completely determines the properties of the CNN. The universe of all CNN genes is called the CNN genome. Many applications from image processing, pattern recognition, and brain science 1

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

2

CNN: A Paradigm for

Complexity

can be easily implemented by a CNN "program" defined by a string of CNN genes called a CNN chromosome. T h e first new result presented in this exposition asserts that every Boolean function of the neighboring-cell inputs can be explicitly synthesized by a CNN chromosome. This general theorem implies that every cellular automata (with binary states) is a CNN chromosome. In particular, a constructive proof is given which shows that the gameof-life cellular a u t o m a t a can be realized by a CNN chromosome made of only three CNN genes. Consequently, this "game-of-life" CNN chromosome is a universal Turing machine, and is capable of self-replication in the Von Neumann sense [Berlekamp et al., 1982]. One of the new concepts presented in this exposition is that of a general­ ized cellular automata (GCA), which is outside the framework of classic cel­ lular (Von Neumann) a u t o m a t a because it cannot be defined by local rules: It is simply defined by iterating a CNN gene, or chromosome, in a "CNN D O L O O P " . This new class of generalized cellular automata includes not only global Boolean maps, b u t also continuum-state cellular automata where the initial state configuration and its iterates are real numbers, not just a finite number of states as in classical (von Neumann) cellular automata. Another new result reported in this exposition is the successful implemen­ tation of an analog input analog output CNN universal machine, called a CNN universal chip, on a single silicon chip. This chip is a complete dynamic array stored-program computer where a CNN chromosome (i.e. a CNN al­ gorithm or flow chart) can be programmed and executed on the chip at an extremely high speed of 1 Tera (10 12 ) analog instructions per second (based on a 100 x 100 chip). T h e CNN universal chip is based entirely on nonlinear dynamics and therefore differs from a digital computer in its fundamental operating principles. Part II of this exposition is devoted to the important subclass of au­ tonomous CNNs where the cells have no inputs. This class of CNNs can exhibit a great variety of complex phenomena, including p a t t e r n formation, Turing patterns, knots, autowaves, spiral waves, scroll waves, and spatiotemporal chaos. It provides a unified paradigm for complexity, as well as an alternative paradigm for simulating nonlinear partial differential equations ( P D E ' s ) . In this context, rather than regarding the autonomous CNN as an approximation of nonlinear P D E ' s , we advocate the more provocative point of view t h a t nonlinear P D E ' s are merely idealizations of CNNs, because while nonlinear P D E ' s can be regarded as a limiting form of autonomous CNNs, only a small class of CNNs has a limiting P D E representation. Part III of this exposition is rather short but no less significant. It contains in fact the potentially most important original results of this ex­ position. In particular, it asserts that all of the phenomena described in the

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Introduction

3

complexity literature under various names and headings (e.g. synergetics, dissipative structures, self-organization, cooperative and competitive phenom­ ena, far-from-thermodynamic equilibrium phenomena, edge of chaos, etc.) are merely qualitative manifestations of a more fundamental and quantita­ tive principle called the local activity dogma. It is quantitative in the sense that it not only has a precise definition but can also be explicitly tested by computing whether a certain explicitly defined expression derived from the CNN paradigm can assume a negative value or not. Stated in words, the local activity dogma asserts that in order for a non-conservative system or model to exhibit any form of complexity, such as those cited above, the associated CNN parameters must be chosen so that either the cells or their couplings are locally active.

What is a CNN?

5

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

1. W h a t is a C N N ? CNN is an acronym for Cellular Neural Network [Chua & Yang, 1988], a multi-disciplinary research area, 1 with broad applications in image and video signal processing, robotic and biological visions [Werblin et ai, 1994, 1996], and higher brain functions [Roska et ai, 1996; Orzo et ai, 1996]. More recently,2 it has also been used as a paradigm for generating static and dynamic patterns, autowaves, spiral waves, scroll waves and spatio-temporal chaos [Chua et ai, 1996] with diverse applications in image and video signal processing [Roska &; Vandewalle, 1994]. Since these latter applications are broader and not necessarily related to neural networks, it may be more appropriate to decode CNN as Cellular Nonlinear Networks in such applications. Definition. CNN. A CNN is any spatial arrangement of locally-coupled cells, where each cell is a dynamical system which has an input, an output, and a state evolving according to some prescribed dynamical laws [Chua k, Roska, 1997]. Although neither the "cells", nor the "coupling laws", are required to be spatially invariant in the above definition, for the sake of concreteness we 1

For more details, see the following special issues and proceedings of workshops devoted exclusively to CNN: • Special issue on cellular neural networks, International Journal of Circuit Theory and Applications, Vol. 20, Sept./Oct. 1992. • Special issue on cellular neural networks, IEEE Transactions on Circuits and Systems I and II, Vol. 40, Mar. 1993 (in two separate issues). • Proceedings of the IEEE International Workshop on cellular neural networks and their applications, Budapest, Hungary, Dec. 1990. IEEE Catalog Number No. 90TII0312-9 and ISBN 951-721-239-9. • Proceedings of the IEEE International Workshop on cellular neural networks and their applications, Munich, Germany, Oct. 1992. IEEE Catalog Number No. 92TH0498-6 and ISBN 0-7803-875-1. • Proceedings of the IEEE International Workshop on cellular neural networks and their applications, Rome, Italy, Dec. 1994. IEEE Catalog Number No. 94TH0693-2 and ISBN 0-7803-2070-0. • Proceedings of the IEEE International Workshop on cellular neural networks and their applications, Seville, Spain, Jun. 1996. IEEE Catalog Number No. 96TH8180 and ISBN 0-7803-3261. • Proceedings of the IEEE International Workshop on cellular neural networks and their applications, London, England, April 1998. IEEE Catalog Number No. 98TII8359 and ISBN 0-7803-4867-2. 2

See also the special issue on "Nonlinear waves, patterns, and spatio-temporal chaos," IEEE Transactions on Circuits and Systems I and II, Vol. 42, Oct. 1995 (in 2 separate issues).

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

6

CNN: A Paradigm for

Complexity

will assume a translation-invariant (homogeneous) three-dimensional lattice CNN architecture throughout this paper, as shown in Fig. 1, whose cutout view in Fig. 2 shows a typical cell Cijk along with all neighboring cells CQ/37 lying inside a sphere of influence S^*. Mathematically, each cell C^k at location (i, j , k) in the above lattice is a dynamical system whose states evolve according to some prescribed state equations, or in the most abstract setting, as a semi-group, whose dynamics are coupled only among the neighboring cells lying within some prescribed sphere of influence Sijk, centered at (i, j , k). A. Isolated

Cell

Except for some examples in Part II, we will consider mostly the twodimensional case and use a double subscript system to present all results. T h e symbol and relevant variables for an isolated 3 cell for a two-dimensional CNN are shown in Fig. 3(a). Observe that in general four variables are as­ sociated with each isolated cell: 1. 2. 3. 4.

input Uij G Ru threshold Zy € Rz state x ^ e Rx output yij G Ry

In this paper, we will usually assume for simplicity t h a t z ^ is a constant scalar, and all other variables are functions of the continuous4 time t. Given any initial state Xij(to) at t = to, any threshold Zij(t), and any input Ujj(i), the state Xjj(i) of each isolated cell C^ is assumed to evolve for all t > to via a state equation X

ij

=

H?1XU> ziji Uijl

i = l, 2 , . . . , M ; j = l, 2,...,N

(1)

where the "dot" denotes the time derivative and fij [•] denotes a mathemat­ ical operator acting on Xjj(i), Zjj(i) and Uij(t). For example, Eq. (1) may be a nonlinear differential equation with a time delay T, as in x

t j = / u f x i j . z*j> UU'] = fij(*ij(t

~

T

) , zij{t),

Uij(*))

(2)

It can also be a nonlinear integral of any complexity, e.g. Volterra and Wiener operators [Boyd et al., 1982; Boyd & Chua, 1985]. 3

A CNN cell is said to be isolated if it is not coupled to any other cell. Although our "semi-group" definition of CNN includes discrete-time CNNs as a special case, only continuous-time CNNs are considered in this paper.

4

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

What is a CNN?

• • • • • • • • • •

• • • • • • • • • •

• •• •• • • . • • . _.. • • • • •••••••••••■•••< • • •

t • •

• • • • • • •

• • • • • • •

• ••••••••••••••••i

7

*t*!|*t*!*!* •,•,•,•„•,'

•• • • # • • • • • • • • • • • • • 1 * , * . * „ * , * . •• • • • • • • • • • • • • • • • • i * . * . * , * , * • • •• • • # • • • • • • • • • • • • • 1 • , •. • •

Fig. 1. A three-dimensional CNN showing cells located at the lattice sites. The couplings between cells are not shown.

AiiSiiS > • • • • • • • ■

> > » >

• • • •

• • • •

• • • •

• • • • • • ' • • • • • • • • • • • • • • • • • •

•••••••••••••• ••••••••••••••a » • • • • • ■ • • • • • • • •

~

Sphere of influence S^ of cellCp containing all cells C^lE Sp Fig. 2.

A cutout view which exposes an inner cell CY,* and its sphere of influence St]k-

8

CNN: A Paradigm for

Complexity

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Isolated

with input

Cell

C.j

u..

Input Threshold •

H

State x. . 1.1

Output y: IJ

(a)

(b) Fig. 3.

Symbol of an isolated CNN cell and its associated state and output equations.

What is a CNN?

9

In this paper we will assume for simplicity that all cells are identical and fij [•] is an ordinary function so that we can delete the subscripts and write Eq. (1) simply as a non- autonomous system of ordinary differential equations

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

*y(t) = f{Xij(t), *ij(t), Uij(t))

(3)

In general, the output yij(t) may be any function of Xij(t), Zij(t) and yij(t), and possibly their time derivatives of any order. In this paper, we will assume that yij(t) depends only on Xij(t); namely yij(t) = gtjfojit))

(4)

In fact, in many cases (see Part II), the output of interest often coincides with the state, yij{t) = £ij(t), so that gij(-) is simply the identity function. In this paper, an isolated CNN cell with a time-varying input is defined by the non-autonomous state equation (3) and the output equation (4), as shown in Fig. 3(b). Example. Standard Isolated CNN Cell. The standard isolated CNN cell [Chua & Yang, 1988] used most widely in the literature, and which has been implemented in virtually all existing CNN silicon chips, is defined as follows: State equation of standard isolated cell:5 dx'' -£- = ~xij + aijf(xij)

+ bijUij + Zij

(5)

where o^j and bij are weighting coefficients, and

f(xij)

= j f l S t f + l\ - \Xi3 ~1\)={

1,

Xij > 1

x ijt *iji — 1,

x ^ '1 \xij\ < Xij < —1

(6)

Output equation of standard isolated cell: yij = f(xij) 5

(7)

We have abused our notation by using /(•) to denote both the "state equation" in Eq. (3), and the "output equation" in Eq. (6). The content will make it clear which one it is intended for.

10

CNN: A Paradigm for

Complexity

B. Sphere of influence Each CNN cell C{j is, by definition, coupled locally only to those neighbor cells which he inside a prescribed sphere of influence Sij(r) of radius r, where

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Sij(r) = {Cu : max(|fc - i\, \l - j\) < r, 1 < k < M, 1 < I < N} We will usually delete r from S{j(r) and simply write Sij to avoid clutter. For the rectangular array, shown in Fig. 4(a), Sij has a radius r = 1, and will usually be called a neighborhood of radius 1, or a 3 x 3 sphere of influence. In this case, Cij is coupled only to its eight nearest neighbor cells Cki as shown in Fig. 4(a), where (fc, I) = (i + 1, j + 1), (i + 1, j), ( i + 1 , j - 1), (i, j + 1), (i, j - 1), (i -l,j + 1), (i - 1, j), and (i - 1, j - 1). A 5 x 5 sphere (corresponding to r = 2) of influence would enlarge the coupling to 24 cells, as shown in Fig. 4(b). Unless otherwise stated, we will assume a 3 x 3 sphere of influence throughout this paper. C. Local couplings In the most general case, both the input Uki and the output yu of all neigh­ bor cells belonging to Sij are coupled to cell Cij, as depicted schematically in Figs. 5(a) and 5(b), respectively, where it is sometimes convenient to interpret each cell in this figure as a neuron whose state dynamics is cou­ pled to neighboring neurons through "synapses". This interpretation may be considered as an abstract model of real neurons. In Fig. 5(a), the input Uki of each neighbor cell Cki is "sensed" by a synapse (denoted by an adjacent pair of small yellow-green triangles) which then injects a weighted contribution bkiUki into the cell Cij so that the total contributions from the eight neighbor cells is given by 6 B u

( ij) =

J2

hklUkl

(8)

kleSij,kljtij

The symbol kl G Sij under the summation sign in Eq. (8) denotes all cells Cki lying inside Sij. Since the input u^i is an external signal (it is not generated by the dynamics of cell Cki), by definition, we will refer to the eight synapses in Fig. 5(a) as the "control" or "feed-forward" synapses. In Fig. 5(b), the output y^i of each neighbor cell Cki is "sensed" by another synapse whose weighted sum A

(Vij) =

^2

aklVkl

(9)

kleSi:j,kljtij 6

To avoid ambiguity, bki in Eq. (8) should be written bijtki for a space-variant CNN, and bk-i,i~j for a space-invariant CNN. Similar remarks apply to a/tj in Eq. (9).

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

What is a CNN?

Fig. 4. (b). 11

&&fflfflWN&ffi X''"-^i^

jtj&.^'jtfi

t^i

(a)

(b)

A CNN cell CV, with a 3 x 3 sphere of influence (a), and a 5 x 5 sphere of influence

ro

(a) 0>) Fig. 5. The center cell Cij receives a weighted control (feedforward) signal 6*111*1 in (a), and a weighted feedback signal akiVki in (b), from each neighbor cell Cki-

;.jas"MWSV"-SHi

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

What is a CNN?

13

from the eight neighbor cells are also applied to cell Cij. Since y^i is gener­ ated by the dynamics of cell CM, and is then fed back to its neighbor cells, including itself, we will refer to the eight synapses (yellow-green triangles) in Fig. 5(b) as "feedback synapses". One could of course substitute the two linear coupling laws defined by Eqs. (8) and (9) by nonlinear operators. Unless otherwise specified, only the linear coupling laws Eqs. (8) and (9) will be used for the standard CNNs in this paper. D. The standard CNN equation By adding the local couplings Eqs. (8) and (9) to the right-hand side of Eq. (5), we obtain the following equations originally proposed in [Chua &: Yang, 1988] for an M x N CNN array with M rows and N columns: State equations of standard C N N : Xij

= -x^ + Zij+ ^2 akiyki + Yl hi^ki

(10)

Output equation of standard C N N : Vij = * = 1, 2 , . . . , M , j = 1,2,...,N, Eq. (6).

f(xij)

(11)

where /(•) is usually defined by

Observe that we have absorbed the terms aijf(x{j) = a^yij and bijUij from Eq. (5) into the summation signs in Eq. (10) so that unlike Eqs. (8) and (9), the indices kl under the summation signs in Eq. (10) now include kl = ij. Consequently each summation term in Eq. (10) contains nine terms for a 3 x 3 sphere of influence, and 25 terms for a 5 x 5 sphere of influence, etc. For an M x N CNN array, the indices i and j would each run from 1 to M and N, respectively, so that Eq. (10) consists of a system of MAT nonlinear ordinary differential equations. Observe that Eq. (10) is not completely defined for cells whose sphere of influence Sij extends outside of the boundary of the array. Consequently, additional boundary conditions must be specified in order for Eq. (10) to be well defined. The three most commonly chosen boundary conditions are:

14

CNN: A Paradigm for

Complexity

(1) Fixed (Dirichlet) boundary condition Here, the state XM of each cell Cki in Eq. (10) which lies outside of the boundary is assigned a fixed constant value.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(2) Zero flux (Neumann) boundary condition Here, the states #*/ of corresponding neighbor cells perpendicular to the boundaries are constrained to be equal to each other. (3) Periodic (Toroidal) boundary condition Here, the first and last rows (resp., columns) of the array are identified, thereby forming a torus. Finally, before Eq. (10) can be simulated in a computer, or implemented in a sihcon chip, the initial state Xjj(0) for all cells must be specified. We will see in Part I of this paper that for image processing applications, the initial states would sometimes be chosen to coincide with the gray-scale intensity (normalized between " - 1 " for pure white and " 1 " for pure black) of the corresponding discretized picture element — called a pixel — of the input image. E. General CNN equations Together, Eqs. (10) and (11) will henceforth be referred to as the standard CNN equation. There are many applications, however, including many CNNs to be presented in Part II of this paper, where different cell dynamics and coupling laws are called for. In all cases, An M x N CNN is defined by specifying: 1. State equations of isolated cells (henceforth referred to as the CNN cell dynamics) 2. Cell coupling laws (or CNN synaptic law when used in the context of neural networks) 3. Boundary conditions 4. Initial conditions The complete system of MN nonlinear differential equations for an M x N CNN array obtained by combining the CNN "cell dynamics" and the "CNN coupling laws" assumes the following general form:

What is a CNN?

15

C N N state equation: Xij = J

[Xij, Zij;

ukl,yki),

kl G Sij(r)

(12)

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

where u*/, y ^ denote vectors whose components Uki and yki include all neighbor cells Cki G Sij(r). C N N output equation: Vij = 9{xij) i = l,2,...,M;j

=

(13)

l,2,.. .,N.

where z^j, Uij and J/JJ denote as usual the threshold, input, and output of cell Cjj. Assuming all cells are of nth order, i.e. Xj_, G .ft", then Eqs. (12) and (13) constitute a system of nMN non-autonomous nonlinear equations iij = f(xij, Zij{t), ukt(t), g(xfc/)),

kl € Sij{r)

(14)

i = 1 , 2 , . . . , M; j = 1, 2 , . . . , N. In this paper, we will consider mainly the situation where both the threshold and the input of each cell are con­ stants, i.e. Zij(t) = z^ and Uij(t) = Uij, in which case, Eq. (14) becomes an autonomous system. For image processing applications, it is usually more convenient to recast this autonomous system of nMN differential equations into the matrix form (15)

X = F(X) where XX2

XIN

X21

X22

X2N

XM1

2M2

XMN

X =

/(*n)

f(xn)

/(XIN)

f(x2l)

f(x22)

f(x2N)

F(X) =

f{xM\)

f(x\f2)

/(XMN)

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

16

CNN: A Paradigm for

Complexity

In the case of the standard CNNs defined in Eqs. (10) and (11), where all cells are first-order cells, i.e. x^ G R, we can associate the value of x^ (t) at time t with the intensity of some physical variables, such as temperature of a surface, at the corresponding point in space. For higher-order cells, x^ € Rn where n > 1, each component of Xy(t) can be associated with a different physical variable, or color component in a multi-color picture. For example, we can choose third-order cells x^ 6 R? to code the three fundamental colors, e.g. red, green, and blue, for color-image processing applications.

2. Part I: Standard C N N s 2.1.

Standard

CNNs are uniquely

specified by CNN

genes

An autonomous CNN defined by Eq. (15) is said to be completely stable if, and only if, every trajectory originating from any initial state converges to some equilibrium state of Eq. (15). Part I of this paper will be devoted almost exclusively to the theory and applications of completely stable standard CNNs defined by Eqs. (10) and (11) with constant threshold and constant inputs; namely, Standard CNN: iij = -Hj + Zij+

a

kif(xkl)

Y2

+

fc»6Sy(r)

^2

bkiuki

H€5y(r)

Vij = f(xij) = ^{\xij + 1| - \xij - 1|) i = l,2,...,M,j

= l,2,...,N

(16)

For a space-invariant (homogeneous) CNN with a 3 x 3 sphere of influ­ ence (r = 1), Eq. (16) is completely specified by 19 real numbers; namely, one threshold Zij, nine control (feed-forward) coefficients bki [see Fig. 5(a)] and nine feedback coefficients aki [see Fig. 5(b)]. It is often instructive to specify these 19 numbers in the form of a template, as shown in the top of Fig. 6, where we have opted for a single subscript notation for the bki and aw coefficients to avoid clutter. These two 3 x 3 matrices are called the control (feed-forward) template and the feedback template, respectively, in view of the corresponding "synaptic" interpretation in Fig. 7, obtained by a superposition of Figs. 5(a) and 5(b). An alternate interpretation of the role played by the A and the B templates is depicted in the schematic

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

16

CNN: A Paradigm for

Complexity

In the case of the standard CNNs defined in Eqs. (10) and (11), where all cells are first-order cells, i.e. x^ G R, we can associate the value of x^ (t) at time t with the intensity of some physical variables, such as temperature of a surface, at the corresponding point in space. For higher-order cells, x^ € Rn where n > 1, each component of Xy(t) can be associated with a different physical variable, or color component in a multi-color picture. For example, we can choose third-order cells x^ 6 R? to code the three fundamental colors, e.g. red, green, and blue, for color-image processing applications.

2. Part I: Standard C N N s 2.1.

Standard

CNNs are uniquely

specified by CNN

genes

An autonomous CNN defined by Eq. (15) is said to be completely stable if, and only if, every trajectory originating from any initial state converges to some equilibrium state of Eq. (15). Part I of this paper will be devoted almost exclusively to the theory and applications of completely stable standard CNNs defined by Eqs. (10) and (11) with constant threshold and constant inputs; namely, Standard CNN: iij = -Hj + Zij+

a

kif(xkl)

Y2

+

fc»6Sy(r)

^2

bkiuki

H€5y(r)

Vij = f(xij) = ^{\xij + 1| - \xij - 1|) i = l,2,...,M,j

= l,2,...,N

(16)

For a space-invariant (homogeneous) CNN with a 3 x 3 sphere of influ­ ence (r = 1), Eq. (16) is completely specified by 19 real numbers; namely, one threshold Zij, nine control (feed-forward) coefficients bki [see Fig. 5(a)] and nine feedback coefficients aki [see Fig. 5(b)]. It is often instructive to specify these 19 numbers in the form of a template, as shown in the top of Fig. 6, where we have opted for a single subscript notation for the bki and aw coefficients to avoid clutter. These two 3 x 3 matrices are called the control (feed-forward) template and the feedback template, respectively, in view of the corresponding "synaptic" interpretation in Fig. 7, obtained by a superposition of Figs. 5(a) and 5(b). An alternate interpretation of the role played by the A and the B templates is depicted in the schematic

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Part I: Standard CNNs

17

-1 tei. i I

Fig. 6.

For a standard CNN with a 3 x 3 sphere of influence, only 19 real numbers are

needed to specify its dynamics. These 19 numbers can be specified by [z], | B [ \ A | (top), henceforth called the CNN cloning template, or rearranged into a single row (bottom), henceforth called a CNN gene.

diagram in Fig. 8. Observe that the central element of the [AJ and the |_B templates represents the "self-feedback" coefficient a^ and the self "con­ trol" ("feed-forward") coefficient bij of an isolated cell's state equation (5). More complex, nonlinear and delay-type templates have been introduced and used as well [Chua & Roska, 1993]. To conserve space, we will usually rearrange the 19 real numbers from the three CNN templates into a single one-dimensional tape, or ribbon, with 19 entries, henceforth called a CNN gene and denoted by Q, as shown in the bottom of Fig. 6. If we define the vector fj, = [z bg bs h b^ b5 b4 b3 b2 h OQ ag aj a$ 05 a 4 03 a^ a{\

(17)

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

18

CNN: A Paradigm for

Complexity

Fig. 7. The non-central elements of the | A | and B templates specify respectively the weighting coefficients of the eight feedback synapses depicted on the right of the schematic, and the eight control (feedforward) synapses on the left.

then we can recast Eq. (16) into the following compact matrix form: X = F(X; /x),

X G RnM xRn,

/i e R19

(18)

From the perspective of nonlinear dynamics, it is often useful to think of Eq. (18) as a 19-parameter family of ODEs, and to regard the components of (x as bifurcation parameters. In order to apply the theory of dynamical systems, which has been de­ veloped for vector differential equations [Hirsch & Smale, 1974; Guckenheimer &; Holmes, 1983], let us repack the above system of nM x N matrix

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Part I: Standard CNNs

Input U

19

Output Y

State X

Fig. 8 Schematic diagram showing each cell is influenced only by the input Uki of its eight nearest neighbors via the B template, and by the output J/*I of the same eight nearest neighbors via the | A | template.

differential equations into a system of v x 1 vector differential equations: Standard CNN Vector Equation: x = - x + z + A y + B u

(19)

where x, y, u, z G Rv, A, B € Ru x R", v = nMN, where, to avoid clutter, we have abused our notation by using the same symbols as in Eq. (16). For example, the components x\, xi,... , x„ of x are now indexed by a single subscript, in contrast to the double-script indices used in Eq. (16). Hence, if we choose a row-wise ordering scheme for a 4 x 5 CNN, the correspondence between the variables in Eqs. (16) and (19) is as follows: m *

112

*1

12

U

• ■■

Xl5

121

122

a-

a-

a-

15

X6

X7

*25

■

■ ■

••

*

£10

X31

^32

•0-

*

111

112

X35

•

• • -II•

X15

141

142

*

*

116

X17

X45

•

• a■

X20

(20) Observe that the v x v matrices A and B in Eq. (19) are sparse, if not band (e.g. under row-wise or column-wise packing order) matrices whose nonzero elements corresponds to the nonzero entries from the A and B templates, respectively. It can be proved that under the row-wise, column­ wise, diagonal, or any other linear packing scheme, the v x v matrices A and B in Eq. (19) are symmetric if and only if the templates _A_ and B^ are symmetric, respectively, with respect to the center of the templates.

2.2.

Oscillations

and chaos from standard

CNNs

In spite of its deceptive simplicity, the dynamics associated with the stan­ dard CNN equations (16), or its vector version, Eq. (19), can be extremely complex, even for relatively small M and N, as demonstrated in the

20

CNN: A Paradigm for

Complexity

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(a)

(b)

*00

Fig. 9. The 3 x 4 CNN in Fig. 9(a) with a fixed boundary condition has only 2 degrees of freedom, provided by the state variables xi and X2 of the two interior cells C22 and C23, respectively. The dynamics (vector field) of this CNN can be represented by the interactions of the directed graph in Fig. 9(b) where the two nodes represent yi = / ( x i ) and j/2 = /(£2), respectively. The function /(•) is defined by Eq. (6).

following example o f a n M x J V = 3 x 4 CNN, as shown in Fig. 9(a), where the CNN is characterized by the template 7

z= 0 ,

0 B= 0 0

0 &00

0

0 0 , 0

0 A = oo,-i 0

0

0

and 600} are the coupling (synaptic) coefficients to be specified in the following examples. In addition, let us assume this 3 x 4 7

For the sake of clarity, we have included the boundary cells in counting the size of this CNN. In practice, as well as in the following, the boundary cells are usually not included since they do not satisfy the same dynamics as the rest of the cells in the array.

Part I: Standard CNNs 21 CNN has a fixed boundary condition; namely, Xi*j- = 0 ,

Ui.j. = 0

(22)

where (i*, j*) denotes the coordinates of the boundary cells; namely,

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(»•, j*) € {(1, 1), (1, 2), (1, 3), (1, 4), (2, 1), (2, 4), (3, 1), (3, 2), (3, 3), (3, 4)}

(23)

Substituting template (21) and the boundary condition (22) into Eq. (16), we obtain the following state equations for t h e above 3 x 4 CNN: i 2 2 = -X22 + 000/(222) + a 0l/( a ; 23) + ^00'"22(t) *23 = - ^ 2 3 + a0,-l/(222) + 000/(223) + bwU23(t) where / ( • ) is defined by Eq. (6). In terms of our single-subscript notation, Eq. (24) assumes the simpler form: 21 = - 2 1 4- 000/(21) + 001/(22) + booUi(t) (25a) 22 = - 2 2 + 00,-1/(21) + 000/(22) -I- boou^it) V\ = / ( 2 1 ) (25b) 2/2 = / ( 2 2 ) T h e right-hand side of t h e state equation of the 3 x 4 CNN in Fig. 9(a) can b e represented by t h e directed graph shown in Fig. 9(b), which shows clearly the roles played by t h e coupling coefficients. Note t h a t since u\(t) and U2{t) are prescribed input functions of time, the control (feedforward) coefficient 600 plays only a non-dynamics role of "scaling" t h e inputs. On the other hand, the feedback coefficients {ao,-i, oo,o, 00,1} play a decisive role in the nonhnear dynamical behaviors of Eq. (25). In particular, observe t h a t the "central" feedback coefficient ooo of the |_AJ template provides a self-feedback effect. 8 To study the dynamical behaviors of Eq. (25), let us consider two numer­ ical examples. E x a m p l e 2.1.1. Choose ooo = 2, ao,-i = —001 = 2, and 600 = 0. In this case, Eq. (24) reduces to the autonomous system i i = - 2 1 + 2/(21) - 2 / ( 2 2 ) (26) x2 = - 2 2 + 2/(x2) + 2 / ( x i ) 8

In chemical reactions, the self-feedback term is said to be auto-catalytic if aoo > 0 and fix) > 0.

22 CNN: A Paradigm for Complexity One can prove t h a t Eq. (26) has a unique limit cycle [Zou &: Nossek, 1991a].

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

A typical pair of waveforms (x\(t), X2(t)) corresponding to the initial state (xi(0), £2(0)) — (0-1) 01) is shown in Fig. 10(a). The associated trajectory and limit cycle is shown in Fig. 10(b). Example 2.1.2. Choose aoo = 2, ao,-i = -ao,i = 1-2, boo = 1, ui(t) = 4.04 sin(|i), and U2(t) — 0. In this case, Eq. (24) becomes xx = - x i + 2/(xi) - 1.2/(x 2 ) + 4.04 sin i^-t x 2 = -X2 + 1.2/(xi) + 2/(x 2 ) This second-order non-autonomous CNN has been found numerically by [Zou &: Nossek, 1991b] to be chaotic. In particular, the waveforms of (xi(i), X2(t)) corresponding to the initial state (xi(0), x 2 (0)) — (0.1, 0.1) are shown in Figs. 11(a), and 11(b), respectively, along with their continuous broad power spectra. The corresponding trajectory is given by the strange attractor shown in Fig. 12(a). The associated Poincare map obtained by sampling the attractor in Fig. 11(a) at the basic period T = 4 of the input signal u\ (t) is shown in Fig. 12(b) (called a "Lady's shoe" attractor in [Zou & Nossek, 1991b]).

2.3.

Complete

stability

criteria

for standard

CNNs

Virtually all image processing applications and silicon chip implementations of standard CNNs to date are based on the assumption that neither the oscillation nor the chaotic phenomenon exhibited by the two simple CNN examples from the preceding section can occur. In this section, we will present some mathematical criteria which guarantee such complete stability properties. Theorem 2.3.1. State-Boundedness Criterion. If the function /(•) in the output equation (11) is continuous and bounded, then the state X{j(t) of each cell of a standard CNN is bounded for all bounded threshold and bounded inputs. Proof. Equation (10) can be recast into the form ±ij = -xij + g{t)

(28)

where

g{t) = Zij(t)+ J2 akif(xkl{t))+ J2 hiuki{t) kl&Sij

kleSij

(29)

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Part I: Standard CNNs

23

(a)

(b) Fig. 10. The solutions x\(t)(in red) and X2(t)(in blue) of Example 2.1.1 corresponding to £i(0) = 0.1 and X2(0) = 0.1 are shown in (a). The corresponding trajectory in (b) is seen to converge to a limit cycle.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

24

CNN: A Paradigm for

Complexity

(a)

(b) Fig. 11. (a) The solution x\(t) of Example 2.1.2 and its power spectrum (in red). (b) The solution X2{t) of Example 2.1.2 and its power spectrum (in blue).

Since both Zij(t) and Uki(t) are bounded by hypotheses, there exists finite constant K such that max \g(t)\ < K

(30)

0 1 \Xij\ < 1 \x i},\
0) or down (if w^ < 0) in a vertical direction. Consequently w^ is called an offset level. Since aij > 1, by hypothesis, the curve representing g(xij) versus Xij given by Eq. (42) is an odd symmetric three-segment piecewise-linear curve whose inner segment (green) through the origin has a positive slope equal to a^ — 1. The slope of the two outer segments is equal to —1. Consider first Wij = 0 and an arbitrary initial state x^. Since i,j > 0 when g(xij) is in the upper half plane, and iij < 0 when g(xij) is in the lower half plane, starting from any point on this curve, the trajectory must tend to the right (resp. left) along the direction indicated on the curve in the upper (resp., lower) half plane. The resulting curve in Fig. 13(a), along with its arrowheads, is called a dynamic route [Chua, 1969] since it completely determines the evolution of a trajectory originating from any initial point on the curve. When coupling is added, the resulting value of Wij introduces an offset which shifts the curve from Fig. 13(a) upwards if w^ > 0, or downwards if w^ < 0, at equilibrium. Four such translated dynamic routes are shown in Fig. 13(b) corresponding to 4 values of wxj which lead to 4 qualitatively different translated curves. Although the dynamic routes indicated on the curves in Fig. 13(b) are no longer time invariant but shift up and down con­ tinuously until equilibrium is reached, the location of each stable equilibrium point is always to the left of x^ — — 1, or to the right of x^ — 1. It follows that in all cases, J/JJ = — 1, or j/jj = 1. ■ Since the graphical construction shown in Fig. 13(a) is quite general and useful for understanding the nonlinear dynamics of standard CNNs, includ­ ing those which are not completely stable, let us pause to examine some of the salient properties of the piecewise-linear plot T shown in Fig. 13(a). We will often refer to T as the DP plot9 of an isolated standard CNN cell. Observe that independent of the | A | and [Bj templates, the threshold z and the input u, the DP plot T is odd symmetric, had breakpoints located at x^ = ± 1 , and has a slope equal to —1 at both outer segments. Moreover, 9

DP plot is an acronym for driving-point plot. The concept of a DP plot and its associated dynamic route was first introduced in [Chua, 1969] and has since been used extensively for studying first-order nonlinear circuits [Chua et ai, 1985].

30

CNN: A Paradigm for

Complexity

♦ Xy = g(xy)

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

slope = -1

slope = ay-1 > 0

(a) DP Plot T

Xy = h(3(ij ; Wy) = g(xy)+Wy

(b) Translated DP Plot T(t) Fig. 13. (a) The dynamic route for an isolated cell (w^ = 0) shows an unstable equilib­ rium point at Xij = 0, a stable equilibrium point Q- located at Xij(Q-) < — 1 and a stable equilibrium point Q + located at Xij(Q+) > 1. (b) The dynamic route associated with the shifted DP plot of a coupled cell corresponding to four different offset levels Wij > aij > 0, 0 < Wij < a^, —aij < Wij < 0, and Wij < - a y < 0. In each case, the stable equilibrium state is located at xy < — 1, or at xy > 1.

the slope of the middle segment of T depends only on a^, the center element of the _A^ template. The family of DP plots T(t) in Fig. 13(b) consists of translates of T whose offset level W{j is defined by Eq. (43). For constant threshold z, and constant input u, the offset w^ = Wij(t) is time-varying if and only if ctki ^ 0, kl ^ ij. The time variation of W{j(t) depends on the state Xki(t) of each neighbor cell Cki belonging to the sphere of influence SV? where a,ki ^ 0. Even in the

Part I: Standard CNNs 31 time-varying case, Wij (t) tends t o a constant as soon as the magnitude of all

neighbor states xki exceed unity because

[-1,

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

f{Xkl) =

if.

ij = (z - 8) + &WJ + 2N-i

(47)

38

CNN: A Paradigm for Complexity

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

T h e D P plot T defined by ±ij = g{xij) in Eq. (46b) is shown in Fig. 17(a). Since aw — 0, kl ^= ij, the offset level wij in Eq. (47) does not change with time but depends only on the threshold z, the input U{j, and the number N-x of blue neighbors inside Sij.

A *u=Su(*u>

(a)

r

u

slope = -1

slope = 1|

XnFgiiixi})

slope = -1 (b)

slope = -1

slope = 1 Fig. 17. (a) DP plot T of the Edge Detection CNN. (b) Shifted DP plot V(t) for m, = 1 and JV_i = 1 (upper plot) when there is one blue neighbor (u^i = - 1 ) and for N-i = 0 (lower plot) when all eight nearest neighbors in Sij are red (uki = 1).

Part I: Standard CNNs

39

In particular, Wij = -0.5 +

2N-1,

Wij = - 1 6 . 5 + 2AT_!,

if u^ = 1

(48a)

if u^ = -

(48b)

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

It follows from Eq. (48a) that for u^ = 1,

r>i.5,

wau < \=-0.5,

if JV_! > i

„ if JV_i = 0

(49a) V '

For u^ — — 1, we have w^ < - 0 . 5 ,

for

0 < JV_! < 8

(49b)

The two shifted DP plots T(t) corresponding to Eq. (49a) are shown in Fig. 17(b), where the upper plot applies when N-i > 1 and the lower plot applies when N-i — 0. It follows from the dynamic routes on these plots that y^ = 1 (since xtj(t) —> XQ+ > 1 as t —> oo) if AT_i > 1, independent of the initial state. For JV_i = 0 and assuming Xjj(O) = 0, the dynamic route in Fig. 17(b) shows Xij(t) —> XQ_ < — 1 as t —> oo, and hence y^ = —1. For u^ = — 1, Eq. (49b) implies that the shifted DP plot will always lie below the lower plot in Fig. 17(b). Consequently, for Xi;(0) = 0, Xij(t) —> XQ_ < —1 and hence y^ = — 1. Hence, we have proved analytically that a CNN with the cloning tem­ plate (45a) will perform flawlessly as an Edge Detector for any binary input patterns. ■ 2.5.2.

Corner detection

CNN

Suppose we keep [Aj and [BJ templates in Eq. (45a) fixed but change only the threshold z. How does the "qualitative" property of the CNN change as we tune z? Certainly we cannot expect the CNN to continue functioning as an edge detector. This is a one-parameter bifurcation problem and a complete analytical study similar to the above analysis can be carried out. To conserve space, however, we will consider only the case where the threshold is changed from z = —0.5 to z = —8.5. Let us choose the same input images and the same initial state as in Examples 1 and 2 in the preceding Edge Detection CNN so that we can compare the differences in their respective output images. The result is summarized in the next pages. Observe that instead of extracting the edges, only isolated red pixels (i.e. u^ = 1) which are surrounded by at least five blue pixels (i.e. u^ = —1) are extracted. We call such red pixels convex corners relative to the input image in order to distinguish them from those that bear a "concave" geometry, relative to the input image.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

40

CNN: A Paradigm for Complexity

To compare t h e dynamical evolutions between the Edge Detection CNN and the Corner Detection CNN, we choose the same input pattern in Fig. 16 and show the corresponding CNN Flow P a t t e r n in Fig. 18. To emphasize once again t h a t the task prescription for each CNN must hold for all binary input patterns, we choose a more complicated geometric pattern by Escher and show the corresponding CNN flow pattern in Fig. 19. Observe that although one could identify several red pixels as "corners" in an intuitive sense, only convex corners are extracted in the output, as prescribed by the CNN gene Q for corner detection. A comparison between the two CNN flow patterns in Figs. 16 and 18 re­ veals t h a t the nonlinear dynamics of the corner detector CNN initially works like an edge detector before extracting the corners! Indeed, one intuitive al­ gorithm for extracting the convex corners from the input p a t t e r n in Fig. 18 is to first extract all one-pixel width edges and then find the intersections be­ tween the horizontal and the vertical edges from the "intermediate" pattern. T h e corner detection CNN has learned to combine these two algorithmic operations into a single more efficient continuous dynamical evolution. Our next proposition guarantees t h a t the corner detection CNN will func­ tion as prescribed, for all binary input patterns. P r o p o s i t i o n 2.5.2. For any binary input image U , the corresponding steady state binary output image Y of the Corner Detection CNN consists of all convex corners of U . In particular, assuming Xij(0) = 0, we have: (a) Uij — 1 maps into yij = 1 if, and only if, there exist, at least 5 nearest neighbors with Uki = — 1. (b) u^ = — 1 maps into y^ — — 1 independent of its neighbors. Proof [Chua & Shi, 1991]. Since [A and |_BJ remain the same as in t h e Edge Detection CNN, Eq. (47) is applicable also for the corner detection CNN. Substituting z = —8.5 into Eq. (47), we obtain Wij = - 1 6 . 5 + 8uij + 2N-i

(50)

where N_i denotes as before the number of blue (i.e. u^ = —1) neighbors belonging to t h e sphere of influence S{j. If u^ — 1, then Wij = - 8 . 5 +

2N-i

> 1.5

N-i

< -0.5

if if

> 5

N_! < 5

(51)

41

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Part I: Standard CNNs

Fig. 18. Flow pattern of the corner detection CNN showing the input image (top left) at t = 0 and the evolution of the corresponding output image at t = 0, t\, £2, • • •, (•) are described analytically via the closed form formula11 given in [Chua & Kang, 1977],

11

Any scalar piecewise-linear function y = f(x) with a finite number of breakpoints at x = x i , X2,..., i n , and/or discontinuities at x = xi, X2,..., x n has the closed form representation ao + oix + 6i|x — x i | +&2|x — X2I + ■• + 6 „ | x — x n | + c\ sgn(x - Xi) + C2 sgn(x — x 2 ) + • + c „ s g n ( x - x „ )

(54a)

where A f 1, sgn(x) = i ( -1,

x>0 x < 0

This general result was first discovered and published in [Chua &; Kang, 1977]. coefficients aj, bj, and Cj can be easily calculated as follow: a\

The

(mo + mn)

bj = - (m,

y-D

= 1,2

n

if /(•) is continuous at the breakpoint x = Xj ) - /(x~)],

(54b)

otherwise, j = 1, 2 , . . . , n

n

ao = /(0) - ^2{bj\xj\

- c, sgn(ij-))

Here, m,i denotes the slope / ' ( x ) at segment i, i = 0, 1, 2 , . . . , n, where the left-most segment is labeled segment 0, and the right-most segment is labeled segment n. All segments are labeled consecutively from left to right. A point x = X; is called a breakpoint of /(•) if / ' ( x + ) / f'(x~), where x+ and xj denote the right and left limit of f{xj), respectively. If /(•) is continuoxis at all breakpoints, i.e. / ( x t ) = f(x~), then Cj = 0, j = 1, 2 , . . . , n, and all terms involving sgn(x) are absent in Eq. 54(a).

Part I: Standard CNNs

45

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Au

Fig. 21. The feedforward synaptic weight "a" in the B template (56) is a piecewiselinear function with two step-wise discontinuities at Au = ±0.2, where Au = mj — Uki ■

namely, a = -1.5sgn(A?/ - 1.5) - 1.5

(55)

b = -1.5sgn(Ay + 1.5) + 1.5

In general, both the feedback and the control (feedforward) coefficient in the A and B templates may be nonlinear functions of the state x, input u, and output y. For example, the "Gradient Detection CNN" is described by the cloning template

z:\z*],

a B: a a

a 0 a

a a , a

0 A: 0 0

0 1 0

0 0 0

(56)

where the value of the control (feedforward) synaptive weight "a" is given by the stepwise piecewise-linear function shown in Fig. 21. Using the formula given by Eqs. (54a) and (54b), we obtain the following closed form representation for a ( ) : a = - 2 4- 5\Au + 0.2| - 10|Au| + 5|Au - 0.2| 4- 0.5 sgn(Au + 0.2) - 0.5 sgn(Au - 0.2)

(57)

Group 3 contains some 3 x 3 time-delayed cloning templates, where the feedback and/or the feedforward synaptic weights in Eq. (10) may include a time-delayed contribution via the following time-delayed state equation:

46

CNN: A Paradigm

for

Complexity

iiji*) = -Xij(t) + 2 + 5 3

a

klVkl{t) + XI

kl&Si

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

+ J2 °&Vki(t-T)+

b

klukl{t)

kleSi-

£

bTklukl{t-T)

(58)

where T is some prescribed time delay. To conserve space, the two sets of cloning templates |_A_), |_BJ, and A T B T in Eq. (58) are represented compactly in the gallery via the CNN gene (G, GT), where G contains the coefficients {z, a^/, bki} while GT contains the coefficients {aTt, bj[j}. For example, the "Image Difference Computation CNN" is described by the following time-delayed cloning template with a time delay T = 10:

This cloning template is represented by a CNN gene (G, GT) of twice the original length in the CNN gallery as follow: Q ■ -4.75 0.25 0.25 0.25 0.25 2 0.25 0.25 0.25 0.25

GT

■ 0 -0.25 -0.25 -0.25

-0.25

0 0 0 0 1 0 0 0 0

0.25 - 0 . 2 5 - 0 . 2 5 -0.25

0 0 0 0 0 0 0 0 0

(60) Group 4 contains some CNN cloning templates with a 5 x 5 sphere of influence (r = 2). Such templates are needed for fine-grain and higherresolution applications, especially in modeling higher brain functions, such as illusions. The following list contains all CNN genes exhibited in this gallery. Each CNN gene is listed under one of the four groups (group 1 is divided into three subgroups, group 2 is divided into two subgroups, and group 3 is divided into two subgroups) cited above, along with a brief description of its "genetic"

Part I: Standard CNNs

47

properties, referred to in this gallery as a "task" because each CNN gene is designed to implement a well-defined task. Group 1: Standard (linear) 3 x 3 templates

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Subgroup 1: Binary image processing by uncoupled CNNs (a,ki = 0, kl ^ ij) 2.6.1. Threshold CNN Task: Convert a gray-scale image into a binary (black and white, or red and blue in pseudo-color) image, based on the absolute brightness of image pixels. All image pixels which are brighter than some prescribed threshold level are turned into the background color, the remaining ones are turned into the foreground color. Thresholding is the simplest image segmentation method, and assigns image pixels into one of the two semantic categories: Objects and background. 2.6.2. Horizontal Translation CNN 2.6.3. Vertical Translation CNN 2.6.4. Diagonal Translation CNN Task: Translate an image by one pixel in the horizontal, vertical or diagonal direction, respectively. Image translation is one of the elementary operations performed while evaluating similarity of regions of different images in such applications as image recognition, image compression, artificial intelligence etc. 2.6.5. Point Extraction CNN 2.6.6. Point Removal CNN Task: Detect the presence of "points" (isolated black pixels) in a binary image and either extract (in the former case) or remove (in the latter case) all these points from the image. The main application of "point extraction" is to search for point-size image objects (e.g. defects on printed circuit boards) while the main application of "point removal" is the filtering of an impulse noise. 2.6.7. Logic NOT Operation CNN Task: Invert intensities of all binary image pixels — the foreground pixels be­ come the background, and vice versa. Logic NOT operation can be used as

oo

0 0

0

Fig. 2.6.1.

0 0

0

0

0

0 0

0

|o 2 0

0 0 0

0

Album page for Threshold CNN.

■!!§#*#;«

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

0

1

®

0

0 0

0 0

0 0

•j

Album page for Horizontal Translation CNN

0 1

[0

ol

Fig. 2.6.2.

0 0

0

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

on O

Fig. 2.6.3. Album page for Vertical TVanslation CNN.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

en

\l

0 0

0 0 0

Fig. 2.6.4.

1 0

0 0

oj 0 0

'm^

=1«

Initial SiS M#fS

0

|o

mw

Album page for Diagonal TVanslation CNN.

!»64x64

0 0 0

~i#-MfP w *«!

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

to

Fig. 2.6.5.

Album page for Point Extraction CNN.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Fig. 2.6.6.

nun

DUD 0 0

Album page for Point Removal CNN.

^■oTT

■^■1 0|0 0

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

0 1 0 0 0 0

0 -2 0 0 0 0

Fig. 2.6.7.

533

Album page for Logic "Not" Operation CNN.

0 0 0

0 0

oj

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Part I: Standard CNNs

55

an element of Boolean Logic algorithms which operate in parallel on data, represented in the form of a two-dimensional array. 2.6.8. Logic AND Operation CNN 2.6.9. Logic OR Operation CNN CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Task: Perform a pixel-wise logic AND operation (logic OR operation, respec­ tively) on corresponding elements of two binary images. As in the previous case, both operations can be elements of some Boolean Logic algorithms which operate in parallel on data arranged in the form of images. 2.6.10. Edge Detection CNN 2.6.11. Corner Detection CNN Task: Extract convex corners from a binary image. Extraction of convex corners is useful for obtaining a compact representation for images which contain various shapes and which are to be compressed. 2.6.12. Right Edge Detection CNN Task: Extract from a binary image only the objects' boundary pixels (only the object's right boundary pixels, in the latter case). Edge extraction is one of the main operations performed in processing of binary images. 2.6.13. Erosion CNN Task: Perform an "erosion" operation on a given binary image. Image erosion is a mathematical morphology operation, which transforms an image in a way determined by the form of the so-called "structuring element". In the simple case, a binary image erosion can be seen as peeling-off these objects' pixels which can be "reached" by a structuring element moving along the objects' boundary. For each prescribed "erosion structuring element", there is a corresponding erosion CNN gene. Image erosion is an important element of nonlinear image filtering methods. 2.6.14. Dilation CNN Task: Perform a "dilation" operation on a given binary image. Image dilation is a morphological complement of the erosion operation. In the simple case, a dilation of a binary image can be seen as appending to image objects all those background pixels, which can be reached by the chosen structuring

o]

Fig. 2.6.8.

| 0 0 ol 0 2 0 l o 0 ©1

Album page for Logic "And" Operation CNN.

tl: ArraySte>30x48

°J

0 1 0 1 00

0 0

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

0

o o o

°_J!

Fig. 2.6.9. Album page for Logic "Or" Operation CNN.

0 1 0 0 0 o!

0 0

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

oo

Fig. 2.6.10.

Album page for Edge Detection CNN.

EDGEDSraCTIOR \ if olfatti** Unary i

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

to

Fig. 2.6.11.

Album page for Corner Detection CNN.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

o

Fig. 2.6.12.

lo

0

0 1 1

\o o o o

2. J__2

0__0_ _0

Album page for Right Edge Detection CNN.

ol

-1

oj

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

OS

Six 32

1 1 1 0 1 o\

IM. m. 0 T ol 0

Q\

0 o\

Fig. 2.6.13. Album page for Erosion CNN.

Initial State

lo

H*0 T2

tmCNNslmpfypMbeff* \Ufi1k$ perimeter qfaU Mac* 4

Exanpfe 2: Array Size » 100x100

ripticn: 5JJ

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

05

Fig. 2.6.14.

Album page for Dilation CNN.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Part I: Standard CNNs

63

element moving along the objects' boundary. For each prescribed "dilation structuring element", there is a corresponding dilation CNN gene.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Subgroup 2: Binary image processing by coupled CNNs (aw 7^ 0 for at least one kl ^ ij) 2.6.15. Shadow Projection CNN Task: Fill an image with the background color along some chosen direction of projection (horizontal, vertical or diagonal) starting from those objects' pixels which are closest to the projection source. Shadow projection is an element of several image analysis operations, e.g. object labeling procedure. 2.6.16. Horizontal Hole Detection CNN 2.6.17. Vertical Hole Detection CNN 2.6.18. Diagonal Hole Detection CNN Task: Detect the number of holes in a binary image, in the horizontal, vertical or diagonal directions, respectively. A "hole" is a set of adjacent background pixels which form a continuous stripe in either of the above mentioned di­ rections. Hole detection CNNs produce information which is very useful for several image processing applications, e.g. character recognition. 2.6.19. Diagonal Line Detection CNN Task: Extract from a binary image all one-pixel wide diagonal lines. Lines with other orientation can be extracted by related CNN genes. Such elementary operations are performed by dedicated groups of neurons in the human brain. 2.6.20. Selected Objects Extraction CNN Task: Extract marked objects from a binary image. To mark objects, another binary image of the same size as the original one, containing a set of markers, is used. As a result of the operation, all those objects containing pixels which coincide with the markers are extracted. Selected objects extraction is a useful element of the "divide-and-conquer" image analysis strategy, since it provides a means for separate analysis of selected simpler objects.

OS

^t

ol

0

l 0 0 2 0 0

\i Oj

2

0

/irifefi

Fig. 2.6.15. Album page for Shadow Projection CNN.

0 0 0 2 0 0

SHADOW PROJECTION CNN

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

2 0

0

0

0 1 0

0 -1

Album page for Horizontal Hole Detection CNN.

o

0

Fig. 2.6.16.

0

0

ol

0 0

0 0

- .;•-,: ■■■-.viT---5-^T

s**5pi. ,.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

0 0

0

m?—

0 0 0

L°

0 0

1 2 -1 0 0 0

Fig. 2.6.17. Album page for Vertical Hole Detection CNN.

— ■

ttfmt Image: arbitrary

1: ArwySto.30x48

o

0

|o

■^vmlmi

VERTICAL BOLE 1 Ipfth

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

0 0 0

Fig. 2.6.18.

0 0

[o 1 0 0 0 2 0 0 -1

o|

Album page for Diagonal Hole Detection CNN.

0

0

0

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

oo

jppsNK

0

0

0

2

Fig. 2.6.19.

0

-2

0

0.75

[• 0.75

2

0.75

0 0

e.75

Album page for Diagonal Line Detection CNN.

i«2»

■2

0

2

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

to

o

Fig. 2.6.20.

Album page for Selected Objects Extraction CNN.

Initials**

&25

OJtS 0.25 fc25

1

02S

0

0

0.25 «t*4

row

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

70

CNN: A Paradigm for

Complexity

2.6.21. Filled Contour Extraction CNN Task:

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Take two binary images of the same size and extract from the former all those regions which fill completely closed contours of the latter image. This operation is useful in some pattern recognition problems. 2.6.22. Global Connectivity Detection CNN Task: Determine whether a given geometric pattern is "globally" connected in one contiguous piece, or is it composed of two or more disconnected compo­ nents. Among many applications, this CNN gene provides a trivial solution to Minsky's hard labyrinth "global" connectivity problem. 2.6.23. Hole-Filling CNN Task: Fill the interior of all closed contours of a given binary image with the foreground color. Hole filling is frequently used in various image processing procedures, such as objects counting, shape analysis, closed contour detec­ tion etc. 2.6.24. Face-Vase Illusion CNN Task: Simulate the well-known visual illusions where some images can be per­ ceived in an ambiguous way, depending on the initial thought or attention. One of the examples of this phenomenon is the face-vase illusion, where the image can be interpreted either as two symmetric faces, or as a vase. Initial "attention" is implemented by specifying, via a second binary pattern, one of the two ambiguously interpreted regions. Subgroup 3: Gray-scale image processing by coupled CNNs 2.6.25. Half-Toning CNN Task: Convert a gray-scale image into a binary image in such a way that the average gray level for corresponding regions of both images is the same. Due to averaging property of the human visual system, these images will appear to be the same when observed from an appropriate distance. Image half-toning is one of the main image compression techniques used for trans­ mitting graphics over low-bandwidth channels, such as e.g. telephone lines (fax machines).

-1

B1^ .

Mjft.*. .,

■K37P

1 5 1 \o 1 0

(J

Fig. 2.6.21. Album page for Filled Contour Extraction CNN,

l E*«npte2: AirftySt £ ■ 300i:300

^^^H

^^m

^^^^B

BBflk*1

-5.25

jjj 1

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

to

-4

0 1

0 | OS

■OS | 0

Fig. 2.6.22. Album page for Global Connectivity Detection CNN.

0.5 §

0.5 1 3

3 1-0.5

1

-0.5 0

0

\ 0 | 0.5

-Oil 0

0

T^M

Aftl

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

A»\

co

Fig. 2.6.23.

Album page for Hole-Filling CNN.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Fig. 2.6.24.

Album page for Face-Vase Illusion CNN.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

-4

0.1 0.07

0.07

. ()

0.1

0J2

0.1

Fig. 2.6.25.

Example2; Array Size = 200x200

»

o!o7

0.1

0.07

-0.1 •0.071

■0.07

Album page for Half-Toning CNN.

1-0.07

-0.1 1.15

-0.1

«iS

fZgli:

iii

Output Image

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

76

CNN: A Paradigm for

Complexity

2.6.26. Inverse Half-Toning CNN

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Task: Transform a binary image into a gray-scale image in a way which pre­ serves the locally averaged intensities of both images. This operation is complementary to the image half-toning CNN gene. 2.6.27. Texture Discrimination CNN Task: Transform textures in an image which originally have nearly equal average gray-levels to related textures with significantly different average gray-level values. This transformation allows for an easy discrimination (and segmen­ tation) between various textures, based on simple thresholding techniques. The segmentation of images containing textures is one of the most difficult and important problems in image processing and is widely used in medical image analysis, aerial photography analysis, geological samples classification, etc. Group 2: Nonlinear 3 x 3 templates Subgroup 1: Binary image processing 2.6.28. Histogram Generation CNN Task: Generate a histogram for each row of a binary image. In case of binary images, a histogram for a given row is represented by a foreground-color stripe which originates at the row boundary and which has a length equal to a number of foreground pixels originally present in this row. If we perform an image segmentation by thresholding some gray-scale image at different threshold levels, then histograms obtained for corresponding binary images reflect a distribution of image pixel intensities. This information can be used several ways, e.g. for estimating the area of different image objects, for an automated evaluation of the image contrast, etc. 2.6.29. Pixel-Wise Parity Detection CNN Task: Determine whether the number of black nearest neighbors of every image pixel, including this pixel itself, is even or odd. This operation is equivalent to the XOR function of nine Boolean variables, and it is performed in parallel for all image pixels.

Fig. 2.6.26.

Initial StaU

0 0

0

0

0

fo 0 0

oj

^':^^"

h

Album page for Inverse-Half Toning CNN.

BJappte2: ArmyShw 3 t t i H 0

I- ^

0.1 032 0.1 0.07 0.1 0.07

0.07 0.1 0.07

*Sfc-

m

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

oc

Fig. 2.6.27.

Album page for Texture Discrimination CNN.

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

:tf^,i>*'*&>

Fig. 2.6.28.

Album page for Histogram Generation CNN.

EJDDEOEEDDDQQEJEEQODD

i*aSBe^a«

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

o

00

Fig. 2.6.29.

[fllKljftlololn

Album page for Pixel-Wise Parity Detection CNN,

Jy8&=*JBMi&:..

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Part I: Standard CNNs

81

2.6.30. Row-Wise Parity Detection CNN

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

Task: Determine whether the number of black pixels in a row is even or odd. Parity detection is a simple technique used for detecting errors which can occur during a binary data transfers. Row-wise parity detection CNN of size m rows by n columns, can simultaneously check for errors in the form of a set of m binary words, each of the length n bits. Subgroup 2: Gray-scale image processing 2.6.31. Contour Extraction CNN Task: Extract objects' contours from gray-scale images, where contours are de­ fined as those image regions where pixel intensities change abruptly. From the human's perception point of view, contours are one of the most impor­ tant elements of perceived scenes, since they provide us with the information on shapes and location of image objects. 2.6.32. Gradient Detection CNN Task: Extract all locations in a gray-scale image (representing some field inten­ sity) where the gradient of the field is smaller than some prescribed threshold value. Gradient detection can be used to search for relatively smooth image areas and the operation of this CNN can be viewed as a complement of the edge detection. Group 3: Time-delayed templates Subgroup 1: Motion related applications 2.6.33. Motion Detection CNN Task: Detect binary image objects moving in a specified direction at a pre­ scribed range of speeds. The Motion Detection CNN can be sensitive to motion in either one of the eight different "compass" directions (N, NE, E, ES, S, SW, W, NW) depending on data on the CNN gene. Motion detection is an important element of video sequence analysis procedures. Similar tasks are performed by different groups of neurons in the human brain.

00 to

BOWrWHBMKTYl

Fig. 2.6.30.

Album page for Row-Wise Parity Detection CNN,

>1» Array Spp3>»48

nEQDEQEEEDLJQQEEGIDOE

wsmsu

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

oo

2:

Fig. 2.6.31.

Initial State

Album page for Contour Extraction CNN.

Ste= 256x256

30x48

&\ M a I a I a 1 a \ 0 \ a | a \ a\a \ 0\ 0\ 0\0\2\ 0\ 0\ 0\0\

a&^~

IP 1 0 0 *

'>..''x 'JLLw

SB??

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

00

Fig. 2.6.32.

Album page for Gradient Detection CNN.

iMtttfSKHi

EnnnnEnnonQQQQQOQDE

sJSJMwff*

CNN: A Paradigm for Complexity Downloaded from www.worldscientific.com by Mr. Chenghen Yang on 05/01/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

oo

t«3T

Fig. 2.6.33. Album page for Motion Detection CNN.

snapshots of the input strmm

t=0

atlxll

*s»* .