Hi everyone,

We recently released TTI Eval `text-to-image-eval`, an open-source library for evaluating zero-shot classification models like CLIP and domain-specific ones like BioCLIP against your (or HF) datasets to estimate how well the model will perform.

You can evaluate custom and HuggingFace text-to-image/zero-shot image classification models like CLIP, SigLIP, DFN5B, and EVA-CLIP. The evaluation metrics include Zero-shot accuracy, linear probe, image retrieval, and KNN accuracy.

We built this for ML engineers and developers using CLIP models.

Here's the installation guide if you want to get started: https://github.com/encord-team/text-to-image-eval?tab=readme...

I'd love to hear your thoughts on this. I'm open to contributions and feedback from the community. Thank you.

Webinar from last week on how to fine-tune VFMs, specifically Meta's Segment Anything Model (SAM).

What you'll need to follow along the fine-tuning walkthrough:

Images, ground-truth masks, and optionally, prompts from the Stamp Verification (StaVer) Dataset on Kaggle (https://www.kaggle.com/datasets/rtatman/stamp-verification-s...)

Download the model weights for SAM the official GitHub repo (https://github.com/facebookresearch/segment-anything)

Good understanding of the model architecture Segment Anything paper (https://ai.meta.com/research/publications/segment-anything/)

GPU infra the NVIDIA A100 should do for this fine-tuning.

Data curation and model evaluation tool Encord Active (https://github.com/encord-team/encord-active)

Colab walkthrough for fine-tuning: https://colab.research.google.com/github/encord-team/encord-...

I'd love to get your thoughts and feedback. Thank you.