^j=dZddlmZddlZddlmZddlmZddlm Z ddl m Z m Z m Z dd lmZdd lmZmZmZdd lmZmZdd lmZd dlmZmZmZmZmZmZm Z m!Z!ddl"m#Z#ejHe%Z&GddejNZ(GddeZ)GddeZ*GddeZ+GddeZ,Gdde Z-eGdde!Z.eGdd eZ/ed!"Gd#d$e.Z0gd%Z1y)&zHPyTorch ViViT model - modular file inheriting transformer core from ViT.)IterableN)nn)initialization)create_bidirectional_mask)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)PreTrainedModel ViTAttention ViTEmbeddingsViTLayerViTMLPViTModel ViTPoolerViTPreTrainedModel) VivitConfigc`eZdZdZdeffd ZdejdejfdZxZ S)VivitTubeletEmbeddingsay This class turns `pixel_values` of shape `(batch_size, num_frames, num_channels, height, width)` into the initial `hidden_states` (tubelet embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer encoder. The seq_length equals (num_frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). configcNt||j}|j}t |t r|n||f}|j |dz|d|dzz|d|dzz|_||_tj|j|j|||_ y)Nrrr) kernel_sizestride) super__init__ tubelet_size image_size isinstancer num_frames num_patchesrConv3d num_channels hidden_size projection)selfrr%r& __class__s r/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/vivit/modular_vivit.pyr$zVivitTubeletEmbeddings.__init__5s ** && #-j(#CZ*V`Ia   ,q/ 1!} Q/ 1!} Q/ 1  %))   !3!3Vb  pixel_valuesreturnc|jdd}|j|jdjddS)Nrr) transposer-flatten)r.r2s r0forwardzVivitTubeletEmbeddings.forwardEs;#--a3 |,44Q7AA!QGGr1) __name__ __module__ __qualname____doc__rr$torchTensorr7 __classcell__r/s@r0rr,s2 { HELLHU\\Hr1rceZdZdZdeffd Zdejdededejffd Z d d ejd e dejfd Z xZ S) VivitEmbeddingszY Construct the CLS token, position and tubelet patch embeddings for video input. rct|tjt j dd|j |_t||_ |jj}tjt j d|dz|j |_ |jdd|_ |`y)Nr)r#r$r Parameterr<zerosr, cls_tokenrpatch_embeddingsr)position_embeddingsr% patch_size mask_token)r.rr)r/s r0r$zVivitEmbeddings.__init__Ps ekk!Q8J8J&KL 6v >++77 #%<< A{QPVPbPb0c#d  --ab1 Or1 embeddingsheightwidthr3cpt||||||jdz}||jdz}y)Nrr)r#interpolate_pos_encodingrH)r.rJrKrL new_height new_widthr/s r0rNz(VivitEmbeddings.interpolate_pos_encoding[s: (VUCtq11 T__Q// r1r2rNc |j\}}}}}|j|}|jj|dd} t j | |fd}|r||j |||z}ne||jdk7s||jdk7r2td|d|d|jdd|jdd ||jz}|j|}|S) Nr)dimrzInput image size (*z) doesn't match model (z).) shaperFrEexpandr<catrNr& ValueErrorrGdropout) r.r2rN batch_sizer(r+rKrLrJ cls_tokenss r0r7zVivitEmbeddings.forwardas>J>P>P; J fe**<8 ^^**:r2> YY J7Q? ##d&C&CJPVX]&^^J++u8J/J (%9+,Adooa.@-AE$d&>&>>J\\*- r1)F) r8r9r:r;rr$r<r=intrNboolr7r>r?s@r0rArAKsf { 05<<00UX0]b]i]i0 ELLD]b]i]ir1rAc eZdZy)VivitAttentionNr8r9r:r1r0r_r_xr1r_c eZdZy)VivitMLPNr`rar1r0rdrd|rbr1rdc eZdZy) VivitLayerNr`rar1r0rfrfrbr1rfc eZdZy) VivitPoolerNr`rar1r0rhrhrbr1rhcVeZdZUeed<dZdZddgZejdZ y)VivitPreTrainedModelrvivit)videorArfctj||t|tr?t j |j t j |jyy)zInitialize the weightsN)r _init_weightsr'rAinitzeros_rErG)r.modules r0rnz"VivitPreTrainedModel._init_weightssG %%dF3 fo . KK(( ) KK22 3 /r1N) r8r9r:r__annotations__base_model_prefixinput_modalities_no_split_modulesr<no_gradrnrar1r0rjrjs9 !*L9U]]_44r1rjceZdZd dedeffd Zeede dde jdzded e jdzd e e d ef d ZxZS) VivitModelradd_pooling_layercDt||t||_y)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N)r#r$rArJ)r.rryr/s r0r$zVivitModel.__init__s  )&1r1F)tie_last_hidden_statesNr2rNattention_maskkwargsr3c |j||}t|j||}|}|jD] }|||fi|}|j |}|j |j |nd} t || S)a Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```)rN)r inputs_embedsr|N)last_hidden_state pooler_output)rJrrlayers layernormpoolerr ) r.r2rNr|r}embedding_output hidden_stateslayersequence_output pooled_outputs r0r7zVivitModel.forwardsj ??r?s@r0rxrxs2{2t2 E226)..2 ^j''$.^j#'^j t+ ^j +, ^j $ ^j3 ^jr1rxa ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ) custom_introc eZdZdeffd Zee d dejdzdejdzde de e de f d ZxZS) VivitForVideoClassificationrc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jy)NF)ryr) r#r$ num_labelsrxrkrLinearr,Identity classifier post_init)r.rr/s r0r$z$VivitForVideoClassification.__init__ss   ++%@ OUN_N_bcNc"))F$6$68I8IJikititiv r1Nr2labelsrNr}r3c |j|fd|i|}|j}|j|dddddf}d}||j|||jfi|}t |||j |jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```rNNr)losslogitsr attentions)rkrr loss_functionrr rr) r.r2rrNr}outputsrrrs r0r7z#VivitForVideoClassification.forward!sx$.4:: $ 3K$ OU$ "33Aq!9:  %4%%ffdkkLVLD$!//))   r1)NNF)r8r9r:rr$rr r<r LongTensorr]r r r r7r>r?s@r0rrs { 26*.). i ''$.i   4'i #' i +, i  i i r1r)rxrjr)2r;collections.abcrr<rrro masking_utilsrmodeling_outputsrr r processing_utilsr utilsr r r utils.genericrrutils.output_capturingrvit.modeling_vitrrrrrrrrconfiguration_vivitr get_loggerr8loggerModulerrAr_rdrfrhrjrxr__all__rar1r0rsO$ &6bb&@@I5   -   H %HRYYH>*m*Z \  v    )  4- 4 4jjjjjjZ  x "6x  x v Pr1