Assignments for (Q1 = What are the differences between sigmoid and softmax activation functions? and Q2 = How do we know, which activation function should be used in hidden or output layers?) are submitted on discord server.

1:- Sigmoid activation function is used in binary class classification where our output should be either 0 or 1 while softmax activation function is used for multi class classification where our output can be classify into more then 2 classes softmax ensure that sum of probability accross all calsses gives 1. 2:- choosing activation function depends of nature of problem as for hidden layers we mostly use Relu function and for output layer we use Sigmoid/Logistic or Softmax

Sigmoid is primarily used for binary classification, while softmax is used for multi-class classification. Sigmoid produces independent probabilities for each class, making it suitable for binary problems. Softmax ensures that the probabilities across all classes sum to 1, making it suitable for problems with multiple classes.Common activation functions for hidden layers include ReLU (Rectified Linear Unit), Leaky ReLU, and variants like Parametric ReLU (PReLU).For binary classification problems in the output layer, the sigmoid activation function is commonly used. For multi-class classification problems, the softmax activation function is typically used in the output layer to generate a probability distribution across multiple classes.

Assignment 2: How do we know, which activation function should we use in hidden or output layer?Answer:Hidden Layers: ReLU (Rectified Linear Unit): Most common choice for hidden layers in modern neural networks. Computationally efficient, avoids vanishing gradients, and aids in faster training. Use ReLU as a starting point and try other options if needed.Tanh (Hyperbolic Tangent): Similar to sigmoid but with a wider range of output (-1 to 1). Can be useful in some cases but generally less preferred than ReLU due to potential for vanishing gradients.Leaky ReLU: Variant of ReLU that addresses the "dying ReLU" problem by allowing a small, non-zero gradient for negative inputs. Can improve performance in some cases.Parametric ReLU (PReLU): Learns the slope of the negative part of the activation function during training. Can further improve performance over Leaky ReLU.Output Layer: Sigmoid: Binary classification (two possible classes). Outputs a probability between 0 and 1 for each class. Softmax: Multi-class classification (more than two classes). Outputs a probability distribution over all classes, where the probabilities sum to 1. Linear: Regression tasks are where you want to predict a continuous value.

Assignment 1: What are the differences between SoftMax and sigmoid functions? Answer:1. Number of Outputs: Sigmoid: Produces a single output value between 0 and 1, often used for binary classification (e.g., predicting whether an email is spam or not). Softmax: Produces multiple output values that sum up to 1, representing probabilities for each possible class in multi-class classification (e.g., predicting the category of an image among several options).2. Mathematical Formulas: Sigmoid: σ(x) = 1 / (1 + e^(-x)) Softmax: σ(x_i) = e^(x_i) / ∑_j e^(x_j) (where j iterates over all possible classes)3. Common Use Cases: Sigmoid: Logistic regression for binary classification. Output layer of artificial neurons for binary decisions. Activation function in deep neural networks. Softmax: Output layer of multi-class classification models. Language modeling is for predicting the next words in a sequence. Machine translation for generating probability distributions over possible translations.4. Output Ranges: Sigmoid: Outputs a single value between 0 and 1. Softmax: Outputs multiple values between 0 and 1, which sum up to 1, representing a probability distribution over classes.5. Saturation: Sigmoid: Saturates at 0 and 1, meaning its gradients become very small for large positive or negative inputs, potentially slowing down learning in neural networks. Softmax: Less prone to saturation, making it more suitable for multi-class classification tasks.In summary: Use sigmoid for binary classification or modeling the probabilities of individual events. Use softmax for multi-class classification or modeling probability distributions over multiple options.

Differences between sigmoid and softmax1.Use Cases:Sigmoid: Primarily used for binary classification problems where there are two classes. Softmax: Used for multi-class classification problems where there are more than two classes.2.Output Range:Sigmoid: Produces output values between 0 and 1 for each neuron independently. Softmax: Produces output values between 0 and 1, but ensures that the sum of all output values across neurons is 1, representing a probability distribution.

Done