Convergence of threshold networks using their dissipative system model

* Correspondence to: I0 . Cem GoK knar, Electronics and Communication Department, Electrics and Electronics Faculty, Istanbul Technical University, Maslak, 80626, Istanbul, Turkey

LETTER TO THE EDITOR

Convergence of threshold networks using their dissipative

system model

N. Serap S

0engoKr and I0 . Cem GoK knar*

Electronics and Communication Department, Electrics and Electronics Faculty, Istanbul Technical University, Maslak, 80626, Istanbul, Turkey

SUMMARY

A new dynamical energy system model representation is given for threshold networks. Inspired by the relation between stability and dissipativeness of dynamical systems, the convergence property of threshold networks is investigated.Using the energy function inherent within the given model a condition, namely the dissipativeness of the dynamical system, necessary and su$cient condition for the convergence of the threshold network to a "xed point, is given. Also, an easy to check inequality is stated to test the convergence of the threshold network. Copyright( 2000 John Wiley & Sons, Ltd.

KEY WORDS: threshold networks; neural networks; dissipativeness; stability

INTRODUCTION

Results on convergence of threshold networks have found applications in various "elds, to name some: binary stack "lters, neural networks and cellular neural networks [1}4]. On the other hand, the relation between stability and dissipativeness of dynamical systems for linear-in control, partially linear and non-linear systems has been studied in continuous time [5}9]. Some results for discrete-time, linear-in control systems are also obtained [10,11]. The aim of this work is to establish a link between the results obtained for convergence properties of threshold networks and the results obtained for stability of dissipative dynamical systems. In the sequel, threshold networks will be shown to have a non-linear discrete-time dynamical system description with an energy function. Using this description some new results, (e.g. &equivalence of dissipativeness' to

&convergence to a "xed point of the threshold network'), inequality (10) on convergence proper-ties, will be obtained based on concepts derived for dissipative dynamical systems. An example will be considered to clarify the conditions obtained for the convergence of threshold networks.

DISSIPATIVE SYSTEMS Given the following discrete dynamical system description:

x(k#1)"f(x(k), u(k)) (1)

y(k)"g(x(k), u(k)) (2)

with consumed energy function

e(K, K0, u, y)"K~1+ k/K0

w(u(k), y(k)) (3)

where w(u, y) is called supply rate in Reference [10], the system is dissipative if and only if for each x0 of state space, EA(x0)(#R, where

EA(x0)": sup_x0C K*0

G

!K~1+

k/0w(u(k), y(k))

H

with the supremum being taken over all K*0 and all trajectories with initial state x0. Since, the use of internal energy function is quite common in investigating dissipativeness [8], its de"nition is given below.

A function EI mapping state space into non-negative real numbers and satisfying

EI(x(K2))!EI(x(K1)))K2+~1 k/K1

w(u(k), y(k)) (5)

for all input-state trajectories Mu(k), y(k)N on [K1, K2] and all 0)K1)K2 is called internal energy function. The system given by (1)}(3) is dissipative if and only if there exists an internal energy function [11]. The concepts given in this section will be used in the following section to obtain some results on the convergence properties of threshold networks.

THRESHOLD NETWORKS Threshold networks are de"ned by the following equalities:

x(k#1)"S(Ax(k)#b) (6)

Here the ith row of S(Ax#b)3Rn, Si is a threshold function performing the non-linear mapping as de"ned in the following:

Si(aix#bi)"

G

where ai and bi denote the ith row of A and b, respectively. This structure is called &general linear threshold network structure' since S()) is de"ned through a linear inequality. Matrix A3Rn]Rn denotes any arbitrary matrix and b3Rn is the constant bias (threshold vector) and x(k)3M!1, 1Nn.

An important problem is the convergence of the network to a "xed point. Since the state space is "nite, it is well known that this network either converges (given an initial state the trajectory converges to a "xed point at most in 2n steps) or oscillates (given an initial state the trajectory converges to a limit cycle in "nite time). In order to obtain some results for the convergence of the network to a "xed point, dissipativeness of the network will be investigated by de"ning the supply rate to be

w(u, y)"!a(AS(y)#b!y)T (AS(y)#b!y) (8)

witha'0.

Given a threshold network by (6) and (7) and the consumed energy function (3) related to w as de"ned by (8) it is obvious that, if the network converges to a "xed point for each initial state then it is dissipative as terms in the summation in (4) will be zero after a "nite number of terms. The opposite is also true as stated in the following proposition.

Proposition. A threshold network given by (6) and (7) and its consumed energy function by (3), and

(8) converges to a ,xed point for each initial state, if and only if the network is dissipative. The &only if ' part being shown above, to show &if ' part, using the de"nition given by (4), the following inequality can be written along trajectories:

EA(x0)*K~1+

k/0aEA(x(k#1)!x(k)E22 ∀x03&, ∀K3J`

(9) Let v(k)":x(k#1)!x(k); v(k)3M!2, 0, 2Nn. If Av(k)O0 in every"nite interval then there must exist a K0 (i.e. after the transient dies) and k1*K0 such that v(k1)O0, Av(k1#nK1)"Av(k1) for all n3J` where K1 is the period of the limit cycle (which the threshold network must enter). Hence, the summation in (9) can not be "nite contradicting dissipativeness.Thus Av(k)"0 for all k after some "nite interval [0, KI ] and if Av(k)"0 implies v(k)"0 after the same interval then the proposition is satis"ed. If not, it will be shown that no limit cycle can be formed outside [0, KI ]. Let x(K3 #j) form a limit cycle for j"1, 2,2, k0, then A(x(KI#j#1)!x(K3 #j))" Av(K3 #j)"0 for j"1, 2,2, k0!1 which implies Ax(K3 #j#1)"Ax(K3#j) for j"1, 2,2, k₀!1 giving in turn S(Ax(K3 #j#1)#b)"S(Ax(K3#j)), i.e. a limit cycle reduces to a "xed point proving the proposition.

The above proposition gives a qualitative result which shows the equivalence between dissipativeness and convergence to a "xed point of the threshold network. Inequality (10), given in terms of threshold network parameters, can be used as a testing criterion for

dissipativeness.

[S2A,b(x)!SA,b(x)]T[S2A,b(x)!SA,b(x)])[SA,b(x)!x]T(I!aATA)[SA,b(x)!x] (10) where SA,b(x)":S(Ax#b) and S2A,b(x)":S(ASA,b(x)#b) with 0(a(1, ∀x3&. Now de"ning

EI(x)":[SA,b(x)!x]T[SA,b(x)!x], writing inequality (10) along trajectories and summing up

both sides from k"0 to K!1 the following inequality can be written:

EI(x(K))!EI(x(0)))!K~1+ k/0

(x(k#1)!x(k))TaATA(x(k#1)!x(k))

which shows that EI( ) ) : &PR` is an internal energy function. So the system is dissipative. As an example consider the threshold network with

A"

C

!2 2

2 !2

D

, b"

C

D

It is easy to show that trajectories starting with initial conditions [1 1]T and [!1 !1]T converge to a "xed point [1 1]T, while the trajectories starting with other initial conditions enter a limit cycle. This system is not dissipative since for initial conditions [1 !1]T and [!1 1]T it is not possible to obtain a "nite value for (4). As expected it is also not possible to "nd 0(a(1 satis"ng inequality (10).

CONCLUSION

In this work the threshold network is considered as a dynamical system for which an an energy function is de"ned. Necessary and su$cient condition for the convergence properties of threshold networks is given using the relation between convergence and dissipativeness. Also an inequality, i.e. (10), is given to check the dissipativeness of the threshold network. These results are obtained using concepts related with dissipative dynamical systems.

REFERENCES

1. Wendt PD. Nonrecursive and recursive stack #ters and their "ltering behavior. IEEE ¹ransactions on Acoustics,

Speech and Signal Processing, 1990; 38:2099}2107.

2. Bruck J, Goodman JW. A generalized convergence theorem for neural networks. IEEE ¹ransactions on Information ¹heory 1988; 34:1089}1092.

3. Harrer H, Nossek JA. Discrete-time cellular neural networks. International Journal of Circuit ¹heory and Applications 1992; 20:453}467.

4. Goles-Chacc E, Fogelman-Soulie F, Pellegrin D. Decreasing energy functions as a tool for studying threshold networks. Discrete Applied Mathmatics 1985;(12):261}277.

5. Hill D, Moylan P, The stability of nonlinear dissipative systems. IEEE ¹ransactions on Automatic Control 1976; AC-21:708}711.

6. Hill D, Moylan P. Stability results for nonlinear feedback systems. Automatica 1977; 13:377}382.

7. Saberi A, Kokotovich PV, Sussmann HJ. Global stabilization of partially linear composite systems. SIAM Journal of

8. Byrnes CI, Isidori A, Willems JC. Passivity, feedback equivalence and the global stabilization of minimum phase nonlinear systems. IEEE ¹ransactions on Automatic Control 1991; AC-36:1228}1240.

9. Wyatt JL, Chua LO, Gannett JW, GoKknar I0 C, Green DN. Energy concepts in the state-space theory of nonlinear

n-ports: part 1 * passivity. IEEE ¹ransactions on Circuits and Systems 1981; CAS-28(1):48}60.

10. Byrnes CI, Lin W. Lossless, feedback equivalence and the global stabilization of discrete-time nonlinear systems.

IEEE ¹ransactions on Automatic Control 1994; AC-39:83}98.

11. Cem GoKknar I0 , S

0engoKr NS. Discrete-time version of Kalman}Yacubovitch}Popov lemma for nonlinear systems.