ó Y[c@sÝddlZddlZddljZddlmZddlmZddlm Z ddl m Z d„Z de fd„ƒYZd e fd „ƒYZd „Zd e fd „ƒYZdefd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZdefd„ƒYZde fd„ƒYZdd lmZdd!lmZed"„Zd#e fd$„ƒYZd%e fd&„ƒYZ dS('iÿÿÿÿN(t downsample(tconv(tconv3d2d(trepeatcCstjd|ƒS(sRectified linear units (relu)i(tTtmaximum(tX((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pytrelu st padding_2DcBseZd„ZRS(c CsÈ||_|\}}}}tjtj|||dfdtjjƒƒ}tj|||gddƒ}tjtj||d|dfdtjjƒƒ} tj| || gddƒ} | |_ dS(Nitdtypetaxisii( tinputttheanotsharedtnumpytzerostconfigtfloatXRt concatenatetoutput( tselfR t data_shapetNstCtHstWstpad_wtx_pad_wtpad_htx_pad_h((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyt__init__s 04(t__name__t __module__R(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR st padding_3DcBseZd„ZRS(cCs*||_|\}}}}}tjtj||||dfdtjjƒƒ}tj|||gddƒ} tjtj|||d|dfdtjjƒƒ} tj| | | gddƒ} tjtj|d||d|dfdtjjƒƒ} tj| | | gddƒ} | |_ dS(NiR R iii( R R R RRRRRRR(RR RRtTsRRRRRRRtpad_dtx_pad_d((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR s 37;(RR R(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR!scCsctjjj|jdƒƒ}|jdddd|d|jƒ}|tj|tj j ƒ}|S(s+p is the probablity of dropping a unit i?Btnitptsize( R ttensortshared_randomstreamst RandomStreamstrandinttbinomialtshapeRtcastRR(trngtlayerR&tsrngtmaskR((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyt_dropout_from_layer/s  %t Conv_3d_LayercBseZddd„ZRS(c Csƒ||_|d|dks#t‚tj|dƒ}|dtj|dƒ}tjd||ƒ} |dkr½tjtj|j d| d| d|ƒdtj j ƒd t ƒ|_ n ||_ |dkrtj|dfdtj j ƒ} tjd | d t ƒ|_n ||_tjd |d |j d |d|ƒ} t| |jjddddƒƒ|_|j |jg|_dS(Niiig@tlowthighR'R tborrowtvaluetsignalstfilterst signals_shapet filters_shapetx(R tAssertionErrorRtprodtsqrttNoneR R tasarraytuniformRRtTruetWRtbRtconv3dRt dimshuffleRtparams( RR/R t filter_shapet image_shapeRERFtfan_intfan_outtW_boundtb_valuest conv_out5D((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR>s&     " (N(RR RAR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR4<stDropout_Conv_3d_LayercBseZdddd„ZRS(gà?c CsVtt|ƒjd|d|d|d|d|d|ƒt||jd|ƒ|_dS(NR/R RJRKRERFR&(tsuperRQRR3R(RR/R RJRKt dropout_rateRERF((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRcs%N(RR RAR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRQbst Sum_3d_LayercBseZdd„ZRS(ic CsŽ||_tj||jd|jd|jd|jd|jdfddƒ}tjd|d|d td d ƒ}||_g|_dS( NiiiiitndimR tdst ignore_bordertmodetsum( R RtreshapeR-Rt max_pool_2dRDRRI(RR tpoolsizet conv_out4Dt pooled_out((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRms  K (ii(RR R(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRTkstLeNetConvPoolLayercBs,eZdZdd„Zd„Zd„ZRS(s&Pool Layer of a convolutional network ic Cs|d|dkst‚||_tj|dƒ}|dtj|dƒtj|ƒ}tjd||ƒ}tjtj|jd| d|d|ƒdtj j ƒd t ƒ|_ tj |dfdtj j ƒ} tjd | d t ƒ|_tjd |jd |j d |d|ƒ} tjd | d|dt ƒ} t| |jjddddƒƒ|_|j |jg|_dS(Niiig@R5R6R'R R7R8R R:RJRKRVRWR=(R>R RR?R@R R RBRCRRRDRERRFRtconv2dRR[RRHRRI( RR/R RJRKR\RLRMRNROtconv_outR^((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRƒs"   "(cCs|jjƒ|jjƒfS(N(REt get_valueRF(R((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyt __getstate__¤scCs0|\}}|jj|ƒ|jj|ƒdS(N(REt set_valueRF(RtstateRERF((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyt __setstate__§s (ii(RR t__doc__RRcRf(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR_€s ! tLogisticRegression_unbalancedcBs/eZdZddd„Zd„Zd„ZRS(s5Multi-class Logistic Regression Class The logistic regression is fully described by a weight matrix :math:`W` and bias vector :math:`b`. Classification is done by projecting data points onto a set of hyperplanes, the distance to which is used to determine a class membership probability. cCs@|d krKtjdtj||fdtjjƒdddtƒ|_n'tjd|dddtƒ}||_|d krºtjdtj|fdtjjƒdddtƒ|_ n'tjd|dddtƒ}||_ t j j t j ||jƒ|j ƒ|_t j|jddƒ|_|j|j g|_d S( s  Initialize the parameters of the logistic regression :type input: theano.tensor.TensorType :param input: symbolic variable that describes the input of the architecture (one minibatch) :type n_in: int :param n_in: number of input units, the dimension of the space in which the datapoints lie :type n_out: int :param n_out: number of output units, the dimension of the space in which the labels lie R8R tnameRER7RFR iN(RAR R RRRRRDRERFRtnnettsoftmaxtdott p_y_given_xtargmaxty_predRI(Rt batch_sizeR tn_intn_outRERF((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRµs*        +cCsDtj|jƒtj|jdƒ|f}||}tj|ƒ S(s•Return the mean of the negative log-likelihood of the prediction of this model under a given target distribution. .. math:: rac{1}{|\mathcal{D}|} \mathcal{L} ( heta=\{W,b\}, \mathcal{D}) = rac{1}{|\mathcal{D}|} \sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \ \ell ( heta=\{W,b\}, \mathcal{D}) :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label Note: we use the mean instead of the sum so that the learning rate is less dependent on the batch size i(RtlogRmtarangeR-tmean(RtytweightsRptorit unbalanced((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pytnegative_log_likelihoodús2, cCsw|j|jjkr<tdd|jd|jjfƒ‚n|jjdƒrjtjtj|j|ƒƒSt ƒ‚dS(sQReturn a float representing the number of errors in the minibatch over the total number of examples of the minibatch ; zero one loss over the size of the minibatch :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label s+y should have the same shape as self.y_predRvRotintN( RURot TypeErrorttypeR t startswithRRutneqtNotImplementedError(RRv((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyterrors=s !N(RR RgRARRzR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRh¬sE CtLogisticRegressioncBs8eZdZddd„Zd„Zd„Zd„ZRS(s5Multi-class Logistic Regression Class The logistic regression is fully described by a weight matrix :math:`W` and bias vector :math:`b`. Classification is done by projecting data points onto a set of hyperplanes, the distance to which is used to determine a class membership probability. cCs|dkrEtjdtj||fdtjjƒddƒ|_n ||_|dkrtjdtj|fdtjjƒddƒ|_n ||_t j j t j ||jƒ|jƒ|_ t j|j ddƒ|_t j ||jƒ|j|_|j|jg|_dS( s  Initialize the parameters of the logistic regression :type input: theano.tensor.TensorType :param input: symbolic variable that describes the input of the architecture (one minibatch) :type n_in: int :param n_in: number of input units, the dimension of the space in which the datapoints lie :type n_out: int :param n_out: number of output units, the dimension of the space in which the labels lie R8R RiRERFR iN(RAR R RRRRRERFRRjRkRlRmRnRotscoreRI(RR RqRrRERF((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR^s  !    +cCs4tjtj|jƒtj|jdƒ|fƒ S(s•Return the mean of the negative log-likelihood of the prediction of this model under a given target distribution. .. math:: rac{1}{|\mathcal{D}|} \mathcal{L} ( heta=\{W,b\}, \mathcal{D}) = rac{1}{|\mathcal{D}|} \sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \ \ell ( heta=\{W,b\}, \mathcal{D}) :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label Note: we use the mean instead of the sum so that the learning rate is less dependent on the batch size i(RRuRsRmRtR-(RRv((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRz•scCs|jd|fS(Ni(Rƒ(RRv((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyt class_score´scCsw|j|jjkr<tdd|jd|jjfƒ‚n|jjdƒrjtjtj|j|ƒƒSt ƒ‚dS(sQReturn a float representing the number of errors in the minibatch over the total number of examples of the minibatch ; zero one loss over the size of the minibatch :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label s+y should have the same shape as self.y_predRvRoR{N( RURoR|R}R R~RRuRR€(RRv((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR·s !N(RR RgRARRzR„R(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR‚Us 7  tLogisticRegression_threscBs8eZdZddd„Zd„Zd„Zd„ZRS(s5Multi-class Logistic Regression Class The logistic regression is fully described by a weight matrix :math:`W` and bias vector :math:`b`. Classification is done by projecting data points onto a set of hyperplanes, the distance to which is used to determine a class membership probability. cCsÿ|dkrEtjdtj||fdtjjƒddƒ|_n ||_|dkrtjdtj|fdtjjƒddƒ|_n ||_t j j t j ||jƒ|jƒ|_ |j dd…dfdk|_|j|jg|_dS( s  Initialize the parameters of the logistic regression :type input: theano.tensor.TensorType :param input: symbolic variable that describes the input of the architecture (one minibatch) :type n_in: int :param n_in: number of input units, the dimension of the space in which the datapoints lie :type n_out: int :param n_out: number of output units, the dimension of the space in which the labels lie R8R RiRERFNig§èH.ÿï?(RAR R RRRRRERFRRjRkRlRmRoRI(RR RqRrRERF((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRØs  !    +"cCs4tjtj|jƒtj|jdƒ|fƒ S(s•Return the mean of the negative log-likelihood of the prediction of this model under a given target distribution. .. math:: rac{1}{|\mathcal{D}|} \mathcal{L} ( heta=\{W,b\}, \mathcal{D}) = rac{1}{|\mathcal{D}|} \sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \ \ell ( heta=\{W,b\}, \mathcal{D}) :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label Note: we use the mean instead of the sum so that the learning rate is less dependent on the batch size i(RRuRsRmRtR-(RRv((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRzscCs|jd|fS(Ni(Rƒ(RRv((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR„-scCsw|j|jjkr<tdd|jd|jjfƒ‚n|jjdƒrjtjtj|j|ƒƒSt ƒ‚dS(sQReturn a float representing the number of errors in the minibatch over the total number of examples of the minibatch ; zero one loss over the size of the minibatch :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label s+y should have the same shape as self.y_predRvRoR{N( RURoR|R}R R~RRuRR€(RRv((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR0s !N(RR RgRARRzR„R(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR…Ïs 6  t HiddenLayercBseZddejd„ZRS(c Csg||_|dkr´tj|jdtjd||ƒ dtjd||ƒd||fƒdtjjƒ}|tj j j kr“|d9}ntj d|dd d t ƒ}n|dkrÿtj|fdtjjƒ} tj d| dd d t ƒ}n||_||_tj||jƒ|j} |dkr?| n || ƒ|_|j|jg|_dS( NR5g@R6R'R iR8RiRER7RF(R RARRBRCR@R RRR(RjtsigmoidR RDRRERFRRlRRI( RR/R RqRrRERFt activationtW_valuesROt lin_output((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRJs(    ! !  N(RR RARttanhR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR†IstDropoutHiddenLayercBseZeddd„ZRS(c Cs\tt|ƒjd|d|d|d|d|d| d|ƒt||jd|ƒ|_dS( NR/R RqRrRERFRˆR&(RRRŒRR3R( RR/R RqRrRˆRStuse_biasRERF((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRzs$N(RR RDRAR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRŒystIntegrate_attrcBseZdZd„ZRS(s5Multi-class Logistic Regression Class The logistic regression is fully described by a weight matrix :math:`W` and bias vector :math:`b`. Classification is done by projecting data points onto a set of hyperplanes, the distance to which is used to determine a class membership probability. cCs:||_||_tj|j|jgddƒ|_dS(NR i(Rxtvec20RRR(RR R%R((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRŒs  (RR RgR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRŽƒs(R((tDownsampleFactorMaxcCso|jdkrtdƒ‚n|j}|jd}tj|jd ƒ}tj|dƒ}tjtjd|tjdgƒ|ƒdƒ}tj ||ddƒ}t |d|d f|ƒ}||ƒ} tjd|jd | jdƒ} tj | | d|jƒ} t t |dƒƒ|d g|dg|dg} | j | ƒ} | jd}tj| jd ƒ}tj|dƒ}tjtjd|tjdgƒ|ƒdƒ}tj | |ddƒ}t d|df|ƒ}||ƒ}tjd| jd |jdƒ} t t |dƒƒ|dg|dg|d g} tj || d|jƒj | ƒS( ss Takes as input a N-D tensor, where N >= 3. It downscales the input video by the specified factor, by keeping only the maximum value of non-overlapping patches of size (ds[0],ds[1],ds[2]) (time, height, width) :type input: N-D theano tensor of input images. :param input: input images. Max pooling will be done over the 3 last dimensions. :type ds: tuple of length 3 :param ds: factor by which to downscale. (2,2,2) will halve the video in each dimension. :param ignore_border: boolean value. When True, (5,5,5) input with ds=(2,2,2) will generate a (2,2,2) output. (3,3,3) otherwise. is%max_pool_3d requires a dimension >= 3iþÿÿÿiitint32RUii(RUR€R-R(R?tshape_padrightR.tjoint as_tensorRZRtlisttrangeRH(R RVRWtvid_dimt frame_shapeRpt new_shapetinput_4DtopRtoutshapetouttshuflt input_timet vid_shapet input_4D_timetouttime((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyt max_pool_3d™s8    #7  #7t PoolLayer3DcBseZdZdd„ZRS(s Subsampling and pooling layer tmaxcCsM|jjtƒƒ|`|dkr7t||ƒ}n tƒ‚||_dS(s7 method: "max", "avg", "L2", "L4", ... R¥N(t__dict__tupdatetlocalsRR£R€R(RR t pool_shapetmethodR((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRâs   (RR RgR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR¤ßst Unpooling3DcBseZded„ZRS(ic Cs…||_||_|jd}|jd}|jd}t||ddƒ}t||ddƒ}t||ddƒ} | |_dS(NiiiR ii(R\RWRR( RR R\RWts1ts2ts3tupsample_axis1tupsample_axis2tupsample_axis3((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyRós     (iii(RR RDR(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyR«òs(!RR t theano.tensorR(Rttheano.tensor.signalRttheano.tensor.nnetRRttheano.tensor.extra_opsRRtobjectRR!R3R4RQRTR_RhR‚R…R†RŒRŽttheano.tensor.signal.downsampleRtFalseR£R¤R«(((sf/cstor/stanford/rbaltman/users/wtorng/DEEP_LEARNING/3DCNN/FSCNN_models/Detect_ATP_sites/code/layers.pyts2    & ,©zz0  F