Leaky ReLU與Identity的使用方式
想請問下方程式碼中,全連接層的部分最後為什麼需要劃分為Leaky ReLU和Identity兩種,以及各自的使用時機?
def local(self, scope, input, in_dimension, out_dimension, leaky=True, pretrain=True, train=True):
"""Fully connection layer
Args:
scope: variable_scope name
input: [batch_size, ???]
out_dimension: int32
Return:
output: 2-D tensor [batch_size, out_dimension]
"""
with tf.variable_scope(scope) as scope:
reshape = tf.reshape(input, [tf.shape(input)[0], -1])
weights = self._variable_with_weight_decay('weights', shape=[in_dimension, out_dimension],
stddev=0.04, wd=self.weight_decay, pretrain=pretrain, train=train)
biases = self._variable_on_cpu('biases', [out_dimension], tf.constant_initializer(0.0), pretrain, train)
local = tf.matmul(reshape, weights) + biases
if leaky:
local = self.leaky_relu(local)
else:
local = tf.identity(local, name=scope.name)
回答列表
-
2020/01/15 下午 08:49楊哲寧贊同數:不贊同數:留言數:
您好 一般來說我們在convolution 層或Fc層後都會接上relu 等nonlinear activation functions ,但有些情況例外,如resnet 中residual block 中的殘差分支,又或是模型的輸出FC層,除此之外linear model也會用identity activation functions 。identity作用在於,返回一個相同的值,但會在graph 中建立節點。