Free download artificial neural networks by b. yegnanarayana

Yegnanarayana- artifical neural networks Thanks a lot Exactly what i needed. Branch: : Electronics Engineering. Yegnanarayana- artifical neural networks Awesm Yegnanarayana- artifical neural networks Thanx a ton!! Branch: : Electrical Engineering. Yegnanarayana- artifical neural networks Thanks for this. Yegnanarayana- artifical neural networks really useful book, thanks for the link! Tags for this Thread artifical , byegnanarayana , engineering forum , faadooengineers , networks , neural. The time now is PM.

However, understanding the development process of these architectures helps us to evolve new architectures tailored to specific issues. At the application level, one can consider two different categories. In one case it may be possible to map the given application onto a neural network model or architecture. We call such situations as direct applications. Simple associative memories, data compression, optimization, vector quantization and pattern mapping fall into the category of direct application.

But in case of problems such as in speech recognition, image processing, natural language processing and decision making, it is not normally possible to see a direct mapping of the given problem onto a neural network model. These are natural tasks which human beings are good at, but we still do not understand how we do them. Hence it is a challenging task to find suitable neural network models to address these problems [Barnden, ; Cowan and Sharp, Review Questions 1.

Give examples for which heuristic search methods of artificial intelligence are applicable. Discuss the developments in artificial intelligence that led to the interest in exploring new models for computing. What is a rule-based expert system?

Why do we say such systems are 'brittle'? Discuss your answer with an illustration. What are the differences in the manner of solving problems by human beings and by machines? Explain the distinction between pattern and data. What are the features of pattern processing by human beings? Explain, with examples, differences between the following pattern recognition tasks: a Association vs classification b Classification vs mapping c Classification vs clustering 14 Introductipn 8.

Explain the following pattern recognition issues with illustrations: a Pattern variability b Temporal patterns c Stability-plasticity dilemma 9.

What are different methods for solving pattern recognition tasks? What is the difficulty with the existing methods for solving natural pattern recognition problems? What are the issues at the architectural level of artificial neural networks? What are the situations for direct applications of artificial neural networks? What is the difficulty in solving a real world problem like speech recognition even by an artificial neural network model? Chapter 1 Basics of Artificial Neural Networks New models of computing to perform pattern recognition tasks are inspired by the structure and performance of our biological neural network.

But these models are not expected to reach anywhere near the performance of the biological network for several reasons. Firstly, we do not fully understand the operation of a biological neuron and the neural interconnections. Moreover, it is nearly impossible to simulate: i the number of neurons and their interconnections as it exists in a biological network, and ii their operations in the natural asynchronous mode.

However, a network consisting of basic computing units can display some of the features of the biological network. In this chapter, the features of neural networks that motivate the study of neural computing are discussed.

A simplified description of the biological neural network is given in Section 1. The differences in processing by the brain and a computer are then presented. In Section 1. Three classical models of artificial neurons are described in Section 1. It is necessary to arrange the units in a suitable manner to handle pattern recognition tasks. The basic training or learning laws for determining the connection weights of a network to represent a given problem are then discussed in Section 1.

The concluding section gives a summary of the issues discussed in this chapter. The description of the biological neural network in this section is adapted from [Muller and Reinhardt, CI. The fundamental unit of the network is called a neuron or a nerve cell.

Figure 1. Treelike nerve fibres called dendrites are associated with the cell body. These dendrites receive signals from other neurons. Extending from the cell body is a single long fibre called the axon, which eventually branches into strands and substrands connecting to many other neurons at the synaptic junctions, or synapses. The receiving ends of these junctions on other cells can be found both on the dendrites and on the cell bodies themselves.

The axon of a typical neuron leads to a few thousand synapses associated with other neurons. Characteristics of Neural Networks 17 The transmission of a signal from one cell to another at a synapse is a complex chemical process in which specific transmitter substances are released from the sending side of the junction.

The effect is to raise or lower the electrical potential inside the body of the receiving cell. If this potential reaches a threshold, an electrical activity in the form of short pulses is generated. When this happens, the cell is said to have fired. These electrical signals of fixed strength and duration are sent down the axon. Generally the electrical activity is confined to the interior of a neuron, whereas the chemical mechanism operates a t the synapses.

The dendrites serve as receptors for signals from other neurons, whereas the purpose of an axon is transmission of the generated neural activity to other nerve cells inter-neuron or to muscle fibres motor neuron.

A third type of neuron, which receives information from muscles or sensory organs, such as the eye or ear, is called a receptor neuron. The size of the cell body of a typical neuron is approximately in the range micrometers pm and the dendrites and axons have diameters of the order of a few pm. The gap at the synaptic junction is about nanometers nm wide.

The total length of a neuron varies from 0. The resulting resting potential of about - 70 mV is supported by the action of the cell membrane, which is impenetrable for the positive Sodium ions. This causes a deficiency of positive ions in the protoplasm. Signals arriving from the synaptic connections may result in a temporary depolarization of the resting potential. This sudden change in the membrane potential causes the neuron to discharge. Then the neuron is said to have fired.

The membrane then gradually recovers its original properties and regenerates the resting potential over a period of several milliseconds. During this recovery period, the neuron remains incapable of further excitation. The discharge, which initially occurs in the cell body, propagates as a signal along the axon to the synapses. The intensity of the signal is encoded in the frequency of the sequence of pulses of activity, which can range fiom about 1 to per second.

The speed of propagation of the discharge signal in the cells of the human brain is about 0. The discharge signal travelling along the axon stops at the synapses, because there exists no conducting link to the next neuron. Transmission of the signal across the 18 Basics of Artificial Neural Networks synaptic gap is mostly effected by chemical activity.

When the signal arrives at the presynaptic nerve terminal, special substances called neurotransmitters are produced in tiny amounts. The neurotransmitter molecules travel across the synaptic junction reaching the postsynaptic neuron within about 0. These substances modify the conductance of the postsynaptic membrane for certain ions, causing a polarization or depolarization of the postsynaptic potential. If the induced polarization potential is positive, the synapse is termed excitatory, because the influence of the synapse tends to activate the postsynaptic neuron.

If the polarization potential is negative, the synapse is called inhibitory, since it counteracts excitation of the neuron. All the synaptic endings of an axon are either of an excitatory or an inhibitory nature. The cell body of a neuron acts as a kind of summing device due to the net depolarizing effect of its input signals. This net effect decays with a time constant of ms. But if several signals arrive within such a period, their excitatory effects accumulate. When the total magnitude of the depolarization potential in the cell body exceeds the critical threshold about 10 mV , the neuron fires.

The activity of a given synapse depends on the rate of the arriving signals. An active synapse, which repeatedly triggers the activation of its postsynaptic neuron, will grow in strength, while others will gradually weaken.

Thus the strength of a synaptic connection gets modified continuously. This mechanism of synaptic plasticity in the structure of neural connectivity, known as Hebb's rule, appears to play a dominant role in the complex process of learning. Although all neurons operate on the same basic principles as described above, there exist several different types of neurons, distinguished by the size and degree of branching of their dendritic trees, the length of their axons, and other structural details.

The complexity of the human central nervous system is due to the vast number of the neurons and their mutual connections. Connectivity is characterised by the complementary properties of convergence and divergence. In the human cortex every neuron is estimated to receive a converging input on an average from about lo4 synapses.

On the other hand, each cell feeds its output into many hundreds of other neurons. The total number of neurons in the human cortex is estimated to be in the vicinity of lo1', which are distributed in layers over a full depth of the cortical tissue at a constant density of about 15 x lo4 neurons per mm2. Combined with the average number of synapses per neuron, this yields a total of about synaptic connections in the human brain, the majority of which develop during the first few months after birth.

The study of the properties of complex systems built of simple, identical units may lead to an understanding of the mode of operation of the brain in its various functions, although we are still very far from such an understanding. Characteristics of Neural Networks 1.

Such a structure is called an artificial neural network ANN. Since ANNs are implemented on computers, it is worth comparing the processing capabilities of a computer with those of the brain [Simpson, For the most advanced computers the cycle time corresponding to execution of one step of a program in the central processing unit is in the range of few nanoseconds.

The cycle time corresponding to a neural event prompted by an external stimulus occurs in milliseconds range. Thus the computer processes information nearly a million times faster. Processing: Neural networks can perform massively parallel operations.

On the other hand, the brain operates with massively parallel operations, each of them having comparatively fewer steps. This explains the superior performance of human information processing for certain tasks, despite being several orders of magnitude slower compared to computer processing of information. Size and complexity: Neural networks have large number of computing elements, and the computing is not restricted to within neurons. The number of neurons in a brain is estimated to be about and the total number of interconnections to be around It is this size and complexity of connections that may be giving the brain the power of performing complex pattern recognition tasks which we are unable to realize on a computer.

The complexity of brain is further compounded by the fact that computing takes place not only inside the cell body, or soma, but also outside in the dendrites and synapses.

Storage: Neural networks store information in the strengths of the interconnections. In a computer, information is stored in the memory which is addressed by its location. Any new information in the same location destroys the old information. In contrast, in a neural network new information is added by adjusting the interconnection strengths, without destroying the old information.

Thus information in the brain is adaptable, whereas in the computer it is strictly replaceable. Fault tolerance: Neural networks exhibit fault tolerance since the information is distributed in the connections throughout the network. Basics of Artificial Neural Networks Even if a few connections are snapped or a few neurons are not functioning, the information is still preserved due to the distributed nature of the encoded information.

In contrast, computers are inherently not fault tolerant, in the sense that information corrupted in the memory cannot be retrieved. Control mechanism: There is no central control for processing information in the brain. In a computer there is a control unit which monitors all the activities of computing. In a neural network each neuron a d s based on the information locally available, and transmits its output to the neurons connected to it.

Thus there is no specific control mechanism external to the computing task. While the superiority of human information processing system over the conventional computer for pattern recognition tasks is evident from the basic structure and operation of the biological neural network, it is possible to realize some of its features using a n artificial network consisting of basic computing units. It is possible to show that such a network exhibits parallel and distributed processing capability.

In addition, information can be stored in a distributed manner in the connection weights so as to achieve some fault tolerance. These features are illustrated through several parallel and distributed processing models for cognitive tasks in [Rumelhart and McClelland, ; McClelland and Rumelhart, ; McClelland and Rumelhart, Two of these models are described briefly in Appendix A.

The motivation to explore new computing models based on ANNs is to solve pattern recognition tasks that may sometimes involve complex optical and acoustical patterns also. It is impossible to derive logical rules for such problems for applying the well known A1 methods. It is also difficult to divide a pattern recognition task into subtasks, so that each of them could be handled on a separate processor. Thus the inadequacies of the logic-based artificial intelligence and the limitations of the sequential computing have led to the concept of parallel and distributed processing through ANN.

It may be possible to realize a large number of simple computing units on a single chip or on a few chips, and assemble them into a neural computer with the present day technology. However, it is difficult to implement the large number of synaptic connections, and it is even more difficult to determine the strategies for synaptic strength adjustment learning. Even with these limitations, ANNs can be developed for several pattern recognition tasks for which it is difficult to derive the logical rules explicitly.

The network connection weights can be adjusted to learn from example patterns. The architecture of the network can be evolved to deal with the problem of generalization in pattern classification tasks. ANNs can also be designed to implement selective attention feature required for some pattern recognition tasks. While C Historical Development of Neural Network Principles 21 the adjustment of weights may take a long time, the execution of pattern classification or pattern recall will be much faster, provided the computing units work in parallel as in a dedicated hardware.

Since information is stored in the connections and it is distributed throughout, the network can function as a memory. This memory is content addressable, in the sense that the information may be recalled by providing partial or even erroneous input pattern. The information is stored by association with other stored data like in the brain.

Thus ANNs can perform the task of associative memory. This memory can work even in the presence of certain level of internal noise, or with a certain degree of forgetfulness.

Thus the short-term memory function of the brain can be realized to some extent. Since information is stored throughout in an associative manner, ANNs are somewhat fault tolerant in the sense that the information is not lost even if some connections are snapped or some units are not hctioning. Because of the inherent redundancy in information storage, the networks can also recover the complete information from partial or noisy input pattern.

Another way of looking at it is that an ANN is a reliable system built h m intrinsically unreliable units. Any degradation in performance is 'graceful' rather than abrupt as in the conventional computers. A remarkable feature of ANNs is that it can deal with data that are not only noisy, but also fuzzy, inconsistent and probabilistic, just as human beings do.

All this is due to the associative and dktributed nature of the stored information and the redundancy in the information storage due to large size of the network.

Typically, the stored information is much less than the capacity of the network. Table 1. In Warren McCulloch and Walter Pitts proposed a model of computing element, called McCulloch-Pitts neuron, which performs a weighted sum of the inputs to the element followed by a threshold logic operation [McCulloch and Pitts, Combinations of these computing elements were used to realize several logical computations.

The main drawback of this model of computation is that the weights are fixed and hence the model could not learn fiom examples. In Donald Hebb proposed a learning scheme for adjusting a connection weight based on pre- and post-synaptic values of the variables [Hebb, Hebb's law became a fundamental learning rule in neural networks literature. In a learning machine was developed by Marvin Minsky, in which the connection strengths could be adapted automatically Basics of Artificial Neural Networks 22 Table 1.

But it was in that Rosenblatt proposed the perceptron model, which has weights adjustable by the perceptron learning law [Rosenblatt, The learning law was shown to converge for pattern classification problems, which are linearly separable in the feature space.

While a single layer of perceptrons could handle only linearly separable classes, it was shown that a multilayer perceptron could be used to perform any pattern classification task. But there was no systematic learning algorithm to adjust the weights to realize the classification task. In Minsky and Papert demonstrated the limitations of the perceptron model through several illustrative examples [Minsky and Papert, Lack of suitable learning law for a multilayer perceptron network had put brakes on the development of neural network models for pattern recognition tasks for nearly 15 years till In s Widrow and his group proposed an Adaline model for a eomputing element and an LMS learning algorithm to adjust the weights of an Adaline model Widrow and Hoff, The convergence of the LMS algorithm was proved.

The algorithm was successfully used for adaptive signal processing situations. The resurgence of interest in artificial neural networks is due to two key developments in early s. The first one is the energy analysis of feedback neural networks by John Hopfield, published in and [Hopfield, ; Hopfield, The analysis has shown the existence of stable equilibrium states in a feedback network, provided that the network has symmetric weights, and that the state update is made asynchronously.

Also, in , Rumelhart et al have shown that it is possible to adjust the weights of a multilayer feedforward neural network in a systematic way to learn the implicit mapping in a set of input-output pattern pairs [Rumelhart et al, al. The learning law is called generalized delta rule or error backpropagation learning law.

About the same time Ackley, Hinton and Sejnowski proposed the Boltzmann machine which is a feedback neural network with stochastic neuron units [Ackley et al, A stochastic neuron has an output function yrhich is implemented using a probabilistic update rule instead of a deterministic update rule as in the Hopfield model.

Moreover, , the Boltzmann machine has several additional neuron units, called hidden units, which are used to make a given pattern storage problem representable in a feedback network.

Besides these key developments, there are many other significant contributions made in this field during the past thirty years. Notable among them are the concepts of competitive learning, selforganization and simulated annealing. Self-organization led to the realization of feature mapping. Simulated annealing has been very useful in implementing the learning law for the Boltzmann machine.

Several new learning laws were also developed, the prominent among 24 Basics of Artificial Neural Networks them being the reinforcement learning or learning with critic. Several architectures were developed to address specific issues in pattern recognition. Some of these architectures are: adaptive resonance theory ART , neocognitron and counterpropagation networks.

Currently, fuzzy logic concepts are being used t o enhance the capability of the neural networks to deal with real world problems such as in speech, image processing, natural language processing and decision making [Lin and Lee, A n ANN consists of interconnected processing units.

The general model of a processing unit consists of a summing part followed by.. The summing part receives N input values, weights each value, and computes a weighted sum. The output part produces a signal from the activation value. The sign of the weight for each input determines whether the input is excitatory positive weight or inhibitory negative weight. The inputs could be discrete or continuous data values, and likewise the outputs also could be discrete or continuous.

The input and output could also be deterministic or stochastic or fuzzy. Interconnections: In an artificial neural network several processing units are interconnected according to some topology to accomplish a pattern recognition task. The output of each unit may be given to several units including itself. The amount of the output of one unit received by another unit depends on the strength of the connection between the units, and it is reflected in the weight value associated with the connecting link.

If there are N units in a given ANN, then at any instant of time each unit will have a unique activation value and a unique output value. The set of the N activation values of the network defines the activation state of the network at that instant.

Likewise, the set of the N output values of the network defines the output state of the network at that instant. Depending on the discrete or continuous nature of the activation and output values, the state of the network can be described by a discrete or continuous point in an N-dimensional space.

A weighted Artificial Neural Networks: Terminology sum of the inputs is computed at a given instant of time. The activation value determines the actual output from the output function unit, i. The output values and. Activation dynamics determines the activation values of all the units, i. The activation dynamics also determines the dynamics of the output state of the network.

The set of all activation states defines the activation state space of the network. The set of all output states defines the output state space of the network. Activation dynamics determines the trajectory of the path of the states in the state space of the network.

For a given network, defined by the units and their interconnections with appropriate weights, the activation states determine the short term memory function of the network. Generally, given an external input, the activation dynamics is followed to recall a pattern stored in a network. In order to store a pattern in a network, it is necessary to adjust the weights of the connections in the network.

The set of all weights on all connections in a network form a weight vector. The set of all possible weight vectors define the weight space. When the weights are changing, then the synaptic dynamics of the network determines the weight vector as a function of time. Synaptic dynamics is followed to adjust the weights in order to store the given patterns in the network. The process of adjusting the weights is referred to as learning.

Once the learning process is completed, the final set of weight values corresponds to the long term memory function of the network. The procedure to incrementally update each of the weights is called a learning law or learning algorithm. Update: In implementation, there are several options available for both activation and synaptic dynamics.

In this case, the activation values of all the units are computed at the same time, assuming a given output state throughout.

From the activation values, the new output state of the network is derived. In an asynchronous update, on the other hand, each unit is updated sequentially, taking the current output state of the network into account each time.

For each unit, the output state can be determined from the activation value either deterministically or stochastically. In practice, the activation dynamics, including the update, is much more complex in a biological neural network than the simple models mentioned above. The ANN models along with the equations governing the activation and synaptic dynamics are designed according to the pattern recognition task to be handled.

Models of Neuron 27 was used in the original MP model. Networks consisting of MP neurons with binary on-off output signals can be configured to perform several logical functions FlcCulloch and Pitts, ; Zurada, Excitatory 1 input Inhibitory input Figure 1.

This unit delay property of the MP neuron can be used to build sequential digital circuits. With feedback, it is also possible to have a memory cell Figure 1. In the MP model the weights are fixed. Hence a network using this model does not have the capability of learning. Moreover, the original model allows only binary output states, operating at discrete time steps 1.

The association units perform predetermined manipulations on their inputs. The main deviation from the M P model is that learning i. The desired or target output b is compared with the actual binary output s , and the error 6 is used to adjust the weights.

There is a perceptron learning law which gives a step-by-step procedure for adjusting the weights. Whether the weight adjustment converges or not depends on the nature of the desired input-output pairs to be represented by the model.

The perceptron convergence theorem enables us to determine whether the given pattern pairs are representable or not. If the weight values converge, then the corresponding problem is said to be represented by the perceptron network. Widrow's Adaline model is that, in the Adaline the analog activation value x is compared with the target output b. In other words, the output is a linear fundion of the activation value x.

This weight update rule minimises the mean squared error a2, averaged over all inputs. This law is derived using the negative gradient of the error surface in the weight space. Hence it is also known as a gradient descent algorithm.

This section presents a few basic structures which will assist in evolving new architectures. The arrangement of the processing units, connections, and pattern inputloutput is referred to as topology [Simpson, Artificial neural networks are normally organized into layers of processing units.

The units of a layer are. Connections can be made either from the units of one layer to the units of another layer interlayer connections or among the units within the layer intralayer connections or both interlayer and intralayer connections. Further, the connections across the layers and among the units within a layer can be organised either in a feedforward manner or in a feedback manner.

In a feedback network the same processing unit may be visited more than once. Let us consider two layers F1 and F2 with M and N processing units, respectively.

By providing connections to theJth unit in the F, layer from all the units in the F , layer, as shown in Figures 1. Whenever the input is given to F,, then the Jth unit of F, a Instar b Outstar c Group of instars d Group of outstars e Bidirectional associative memory f Autoassociative memory Figure 1.

Basic Learning Laws 31 will be activated to the maximum extent. Thus the operation of an instar can be viewed as content addressing the memory. In the case of an outstar, during learning, the weight vedor for the connections from the jth unit in F2approaches the activity pattern in F,, when an input vector a is presented at F1.

During recall, whenever the unit j is activated, the signal pattern sjwL, s,wy, Thus the operation of an outstar can be viewed as memory addressing the contents. When all the connections from the units in F1 to F2are made as in Figure 1. This network can be viewed as a group of instars, if the flow is from Fl to F2. On the other hand, if the flow is from F2to F1, then the network can be viewed as a group of outstars Figure 1.

When the flow is bidirectional, we get a bidirectional associative memory Figure 1. If the two layers Fl and F2 coincide and the weights are symmetric, i. Neuronal dynamics consists of two parts: one corresponding to the dynamics of the activation state and the other corresponding to the dynamics of the synaptic weights.

We will discuss models of neuronal dynamics in Chapter 2. In this section we discuss some basic learning laws [Zurada, , Sec. Learning laws are merely implementation models of synaptic dynamics. Typically, a model of synaptic dynamics is described in terms of expressions for the first derivative of the weights.

They are called learning equations. There are different methods for implementing the learning feature of a neural network, leading to several learning laws. Some 32 Basics of Artificial Neural Networks basic learning laws are discussed below. All these learning laws use only local information for adjusting the weight of the connection between two units. The law states that the weight increment is proportional to the product of the input data and the resulting output signal of the unit.

This law represents an unsupervised learning. This is also called discrete perceptron learning law. The expression for Aw, shows that the weights are adjusted only if the actual output si is incorrect, since the term in the square brackets is zero for the correct output. This is a supervised learning law, as the law requires a desired output for each input.

In implementation, the weights can be initialized to any random initial values, as they are not critical. The weights converge to the final values eventually by repeated use of the input-output pattern pairs, provided the pattern pairs are representable by the system. These issues will be discussed in Chapter 4. Hence, This law is valid only for a differentiable output function, as it depends on the derivative of the output function fi.

I t is a supervised learning law since the change in the weight is based on the error between the desired and the actual output values for a given input. Delta learning law can also be viewed as a continuous perceptron learning law. The weights converge to the final values eventually by repeated use of the input-output pattern pairs. The convergence can be more or less guaranteed by using more layers of processing units in between the input and output layers.

The delta learning law can be generalized to the case of multiple layers of a feedforward network. We will discuss the generalized delta rule or the error backpropagation learning law in Chapter 4.

In this case the change in the weight is made proportional to the negative gradient of the error between the desired output and the continuous activation value, which is also the continuous output signal due to linearity of the output function. This is same as the learning law used in the Adaline model of neuron. In implementation, the weights may be initialized to any values. The input-output pattern pairs data is applied several times to achieve convergence of the weights for a given set of training data.

The for any arbitrary training data set. But the Hebbian learning is an unsupervised learning, whereas the correlation learning is a supervised learning, since it uses the desired output value to adjust the weights.

Author Index. Subject Index. Download Sample PDF. Tags: Artificial Neural Networks by B. Ebook Days. MCQs in Computer Science. Fundamentals Of Computers. Fundamentals Of Mobile Computing.

Digital Image Processing And Analysis. IT Tools and Business Systems. Rajendra Prasad, T.