^j d dl mZ d dlmZ d dlZd dlZddlmZm Z ddl
m Z ddl m
Z
mZ e j e Z e ed d
G d d e
Zy)
)UserDict)AnyN )add_end_docstringslogging ) ffmpeg_read)Pipelinebuild_pipeline_init_argsT)has_feature_extractor
has_tokenizerc e Zd ZdZdZdZdZdZ fdZde j e z e z e
z dedee
e ef f fd Zd Zd
d
Zd Zd Z xZS )#ZeroShotAudioClassificationPipelinea
Zero shot audio classification pipeline using `ClapModel`. This pipeline predicts the class of an audio when you
provide an audio and a set of `candidate_labels`.
The default `hypothesis_template` is : `"This is a sound of {}."`. Make sure you update it for your usage.
Example:
```python
>>> from transformers import pipeline
>>> from datasets import load_dataset
>>> dataset = load_dataset("ashraq/esc50")
>>> audio = next(iter(dataset["train"]["audio"]))["array"]
>>> classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-unfused")
>>> classifier(audio, candidate_labels=["Sound of a dog", "Sound of vacuum cleaner"])
[{'score': 0.9996, 'label': 'Sound of a dog'}, {'score': 0.0004, 'label': 'Sound of vacuum cleaner'}]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This audio
classification pipeline can currently be loaded from [`pipeline`] using the following task identifier:
`"zero-shot-audio-classification"`. See the list of available models on
[huggingface.co/models](https://huggingface.co/models?filter=zero-shot-audio-classification).
FTc $ t | di | y )N )super__init__)selfkwargs __class__s /var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/pipelines/zero_shot_audio_classification.pyr z,ZeroShotAudioClassificationPipeline.__init__C s
"6" audiosr returnc $ t | |fi |S )a
Assign labels to the audio(s) passed as inputs.
Args:
audios (`str`, `list[str]`, `np.array` or `list[np.array]`):
The pipeline handles three types of inputs:
- A string containing a http link pointing to an audio
- A string containing a local path to an audio
- An audio loaded in numpy
candidate_labels (`list[str]`):
The candidate labels for this audio. They will be formatted using *hypothesis_template*.
hypothesis_template (`str`, *optional*, defaults to `"This is a sound of {}"`):
The format used in conjunction with *candidate_labels* to attempt the audio classification by
replacing the placeholder with the candidate_labels. Pass "{}" if *candidate_labels* are
already formatted.
Return:
A list of dictionaries containing one entry per proposed label. Each dictionary contains the
following keys:
- **label** (`str`) -- One of the suggested *candidate_labels*.
- **score** (`float`) -- The score attributed by the model to that label. It is a value between
0 and 1, computed as the `softmax` of `logits_per_audio`.
)r __call__)r r r r s r r z,ZeroShotAudioClassificationPipeline.__call__F s . w 1&11r c @ i }d|v r|d |d< d|v r|d |d< |i i fS )Ncandidate_labelshypothesis_templater )r r preprocess_paramss r _sanitize_parametersz8ZeroShotAudioClassificationPipeline._sanitize_parameters_ sI '4:;M4N 0 1 F *7=>S7T 3 4 "b((r c t |t ri|j d s|j d r"t j |d j
}n%t
|d 5 }|j }d d d t |t r t || j j }t |t j s t d t |j dk7 r t# d | j |g| j j d
}|j% | j& }||d < |D cg c] }|j) | }}| j+ |d d }|g|d
< |S # 1 sw Y xY wc c}w )Nzhttp://zhttps://T)follow_redirectsrbz"We expect a numpy ndarray as inputr zNWe expect a single channel audio input for ZeroShotAudioClassificationPipelinept)
sampling_ratereturn_tensorsr )r' padding text_inputs)
isinstancestr
startswithhttpxgetcontentopenreadbytesr feature_extractorr& npndarray TypeErrorlenshape
ValueErrortodtypeformat tokenizer) r audior r finputsx sequencesr) s r
preprocessz.ZeroShotAudioClassificationPipeline.preprocessh sY eS ! *e.>.>z.J %$?GG%& %! FFHE% eU # t'='='K'KLE%, @A A u{{ q mn n''
G4#9#9#G#GX\ (
4::&%5!"?"&&}5 k!nh /%a.K &a.+K $**;{;l; !1 ..
r c |j d }|d d }|j d }|j }t t || d D cg c]
\ }}||d }}}|S c c}}w )Nr rE r )dimc | d S )Nr r )rA s r zAZeroShotAudioClassificationPipeline.postprocess.. s _`ab_c^c r )key)scorelabel)rF softmaxtolistsortedzip) r rK r rE probsscoresrR candidate_labelresults r postprocessz/ZeroShotAudioClassificationPipeline.postprocess s (,,-?@ x(+1% +1V=M1NTc*d
& o 6
s A+)NzThis is a sound of {}.)__name__
__module__ __qualname____doc___load_processor_load_image_processor_load_feature_extractor_load_tokenizerr r4 r5 r2 r+ dictr listr r! rC rL r\
__classcell__)r s @r r r sy : O ! "O#2rzzE 1C 7$ > 2# 2RVW[\_ad\dWeRf 22)8 " r r ) collectionsr typingr r- numpyr4 utilsr r audio_classificationr baser
r
get_loggerr] loggerr r r r rp sa ! . 4
H % ,4W[\]@( @ ^@r