# 🍜 Ramen Broth AI Scoring System

**Grade a pot of broth in 30 seconds  ·  Free up regional managers' attention for what matters**

## Turn 100\+ daily soup videos into a 30\-second "anomaly list"

Staff films as usual  →  AI auto\-scores and flags problems  →  Regional managers focus only on what needs attention

# 🚀 Try the Demo

**Public demo deployed on BytePlus VeFaaS** — just open in your browser, no install needed:

- 🇺🇸 English: [demo \(EN\)](https://sd80t1tno9brblhq062m0.apigateway-ap-southeast-1.apigw-byteplus.com/en)

- 🇨🇳 中文: [demo \(CN\)](https://sd80t1tno9brblhq062m0.apigateway-ap-southeast-1.apigw-byteplus.com/)

Use the language switcher in the top\-right corner of either page\.

## How to use

**Option 1: Browser upload**

1. Open the demo URL

2. "Local Video" tab → pick a MOV / MP4 file

3. Select Brix Level \(11\-14\) \+ Time Slot \(Pre\-open / dinner rush\)

4. Click "Score this broth"

5. 30\-40 seconds later → AI score \+ 4\-dimension reasoning

**Option 2: API call \(have TOS URL\)**

```bash
curl -X POST \
  https://sd80t1tno9brblhq062m0.apigateway-ap-southeast-1.apigw-byteplus.com/api/predict-url \
  -d "video_url=<your-tos-url>" \
  -d "brix=13" \
  -d "time_slot=開店前"
```

**Latency**: browser file upload \~36s \(includes upload \+ Seed 2\.0 Pro inference \+ ML head\); already\-on\-TOS URL \~12s\. Concurrent batch is fine — VeFaaS auto\-scales the function\.

## Sample output

**Real test output** \(a 7\.9 MB MOV uploaded via browser\):

```json
{
  "ai_score": 92,
  "ai_grade": "A",
  "attention_needed": false,
  "attention_prob": 0.002,
  "dim_scores": {
    "Density":  29,
    "Water":    8,
    "Oil":      17,
    "Boiling":  19
  },
  "reasoning": {
    "density": {"score": 29, "label": "Rich", "reason": "Creamy thick broth, matches Brix 13"},
    "water":   {"score": 8,  "label": "Low",   "reason": "Surface far from pot rim"},
    "oil":     {"score": 17, "label": "Slightly heavy", "reason": "..."},
    "boiling": {"score": 19, "label": "Active", "reason": "..."}
  }
}
```

---

# 📊 Headline Metrics

# 74%

**Grade\-match rate**

AI's A / B / C / D grade matches human judgment 74% of the time

# 11\.7

**Mean absolute error**

AI scores are within 11\.7 points of human scores on average \(0\-100 scale\)

# 44%

**Anomaly recall**

Of all genuinely problematic soups, AI flags 44% for review

## 30 sec

Per\-video processing time

## $0\.02

Per\-video scoring cost

## 938

Real videos used for training and evaluation

**Evaluation methodology**: 938 real videos collected over six months \(Nov–Apr\), with 684 carrying human scores\. Stratified 70 / 10 / 20 split into train \(474\) / val \(68\) / test \(137\)\. All metrics are reported on the **137\-record test set, never seen during training or tuning**\.

---

# 💡 What the product does

**Auto Scoring**

Converts each soup video into a **0\-100 AI score** plus an A / B / C / D grade tag\.

**Anomaly Alert**

Intelligently flags soups that need review — managers **focus only on flagged ones**, saving 3\-5× time\.

**Explainable**

Every score comes with text reasoning, broken down across **4 business dimensions**\. Not a black box\.

---

# 🎬 Why it's needed

**Before — fully manual**

- **50\-100\+** videos to review per day

- Manager attention scattered across all stores

- Scoring subjective, hard to compare stores

- Writing feedback is tedious, response is slow

**After — with AI**

- AI auto\-scores in **30 seconds**

- Red / Yellow / Green alerts, **instant focus**

- 4\-dimension objective, comparable across stores

- Reasoning auto\-generated, feedback in seconds

---

# ⚙️ How it works

**Core idea: have the AI score 4 business dimensions independently, then use a supervised ML model to compose the final judgment\.**

The 4 dimensions were not chosen by intuition — they were **derived from keyword analysis on 520 historical human review comments**\.

## Step 1: AI watches the video and scores 4 dimensions

## 30 pts

**Density**

Does visual richness match the declared Brix level?

## 25 pts

**Water Level**

How far is the broth surface from the pot rim?

## 25 pts

**Oil Layer**

Is the oil balance right \(neither too heavy nor too thin\)?

## 20 pts

**Boiling State**

Heat strength and boiling activity

## Step 2: Compose final score

**Key design**: each dimension call sees **3 reference frames** \(high / mid / low anchors\)\. The AI doesn't "judge from feel" — it **scores by comparing to anchors**, yielding stable and reproducible results\.

---

# 🛠️ Technical architecture

**One\-line architecture**: BytePlus Seed 2\.0 Pro multimodal LLM handles visual understanding; XGBoost dual head handles supervised composition\.

## Two\-stage pipeline

|Stage|What we use|What it does|
|---|---|---|
|**Stage 1<br>Visual**|Seed 2\.0 Pro \(VLM\)<br>BytePlus' in\-house multimodal LLM|Runs **4 parallel calls per video**, each focused on one dimension\. Each call sees: video \(fps=1\) \+ 3 reference anchors \+ Brix level \+ time\-slot context|
|**Stage 2<br>Composition**|XGBoost dual head<br>Regressor \+ binary classifier|Feeds 4 dim scores \+ derived features \(**12 total**\) into two XGBoost heads: the **regressor** outputs the continuous score; the **classifier** outputs anomaly probability|

## Key technical decisions

**Why we abandoned PDF's "color / meat\-chunks / bubbles" dimensions**

Measured ρ = \-0\.16, **anti\-correlated with human scores**\. Reason: PDF says "white = 0 points", but tonkotsu broth is naturally creamy white — absolute color carries no information\.

**Why density / water / oil / boiling instead**

NLP keyword stats on 520 review comments showed **these 4 concepts cover 60%** of all reviewer remarks\. "Color" was mentioned in only 1\.3% — discarded\.

**Why ML head instead of fixed weights**

Fixed weights \(30\+25\+25\+20\) plus monotonic calibration capped ρ at 0\.45\. XGBoost learning **non\-linear combination \+ Brix context** pushed ρ to **0\.53**\.

**Why fps=1, not higher**

Measured fps=4 caused **ρ to drop from 0\.44 to 0\.28**\. Extra frames introduce transient noise \(camera shake, momentary reflections\)\. For static\-broth tasks fps=1 is optimal\.

## What the model relies on — feature importance

**Interesting finding**: the model leans most heavily not on any of the 4 VLM\-output scores, but on the **business\-context feature "Brix level"** \(18\.2%\)\. This shows **"domain knowledge \> single visual signal"** — knowing "this should be Brix 13" predicts human scoring better than any single visual judgment\.

|Feature|Importance|Meaning|
|---|---|---|
|**Brix level**|18\.2%|Declared concentration \(strongest prior\)|
|Max\-dim score|11\.4%|Strongest among the 4 VLM dimensions|
|Count of low\-scoring dims|9\.3%|How many dims are in a low quantile \(anomaly signal\)|
|Raw total|9\.2%|Simple sum of 4 dim scores|
|Water\-level score|9\.0%|Water\-level dim \(top business concern, 19% of reviews\)|
|Boiling\-state score|7\.8%|Boiling dim|
|Time slot|7\.7%|Pre\-open vs\. dinner rush|
|Oil\-layer score|6\.6%|Oil dim|

---

# 📥 Output format

For each processed video, the system returns a structured "score card":

## Sample output

**AI Score**: **87**  ·  **Grade**: A \(good broth\)  ·  **Needs review**: No

**4\-dimension breakdown**:

- Density 25 — close to standard Brix\-13 richness

- Water level 22 — surface near pot rim

- Oil layer 18 — even but slightly heavy

- Boiling state 22 — sustained rolling boil

|Field|Meaning / use|
|---|---|
|**AI Score**|0\-100, shown to store managers as the "daily score"|
|**Grade**|A good 80\-100 / B mid 60\-79 / C poor 21\-59 / D very poor 0\-20|
|**Needs review**|True / False — flag for regional manager attention|
|Review confidence|0\-100% probability, threshold tunable \(default 30%\)|
|4 dim scores|Independent per\-dimension scores, pinpoint specific issues|
|Per\-dim reasoning|Plain\-text explanation per dimension, for human reviewer context|

---

# 🛠️ How to use

**Staff films as usual**

Same workflow as today — record soup pot video, tag Brix \+ time slot

**System auto\-scores**

Within 30 seconds, AI score and anomaly flag appear on the dashboard

**Manager reviews on demand**

Only inspect ⚠️\-flagged videos, using AI's reasoning to decide fast

---

# 📈 How we got to ρ = 0\.53 — 5 iterations

**Transparent record**: the system went through 5 versions, each with quantified metric shifts and explicit lessons learned\. This is not a one\-shot product — it's a data\-driven iteration toward optimum\.

|Version|Key change|ρ|MAE|Lesson|
|---|---|---|---|---|
|v1|PDF rubric \(color / meat / bubbles\)|↓ \-0\.16|20\.4|PDF dims anti\-correlate with tonkotsu reality|
|v2|Switched to 4 business dimensions|↑ \+0\.11|22\.1|Direction reversed but resolution poor|
|v3|Strengthened prompts \+ better anchors|↑ 0\.45|13\.5|Ceiling of pure prompt engineering|
|v4|Enabled Pro deep\-thinking mode|0\.44|12\.2|Thinking gives marginal improvement|
|v5|Video fps 1 → 4|↓ 0\.28|13\.7|Extra frames introduce noise — regression|
|**Final**|**XGBoost ML head composing outputs**|**↑ 0\.53**|**11\.7**|Supervised composition broke through ceiling|

**Core insight**: pure prompt engineering capped at ρ = 0\.45\. **The real breakthrough came from feeding VLM outputs \+ business context \(Brix / time\-slot\) into a supervised ML model** — letting the model learn the optimal combination rather than hand\-tuning weights\.

---

# ⚠️ Current limitations

**What the AI cannot do today**

- **Blurry videos / heavy steam**: humans can't read these either, so the AI treats them as "cannot assess"

- **Edge cases with no explicit fault**: when reviewers down\-score "by feel" without writing why, the AI can't see it

- **Cross\-store grading biases**: different regional managers grade with different strictness — not yet modeled

## Data\-noise floor analysis

Of the 9 records in the test set that "should be alerted" \(classes C\+D\), the AI catches 4\. Why the other 5 are unreachable:

- **3 have empty review comments** — human scored low "by feel" without a stated reason; the AI sees a normal\-looking tonkotsu broth

- **2 have blurry / steam\-obscured videos** — even human eyes cannot judge them

So the **theoretical recall ceiling is \~67%** \(≈ 6 / 9\), with the remaining 3 records being intrinsically unrecognizable\. We currently sit at 44%, with \~23 percentage points of headroom — to be closed by multi\-model ensembling and store\-bias modeling\.

The AI agrees with humans on **74% of judgments**; the remaining 26% disagreements cluster on the A / B boundary\. **The system is best deployed as "initial screening \+ anomaly alert" — final judgment stays with the regional manager\.**

---

# 🗺️ Roadmap

**Short\-term \(1\-2 weeks\)**

- **Multi\-model ensemble**: Mini \+ Lite \+ Pro, expecting \+10\-15% on anomaly recall

- **Store\-level bias learning**: add "submitter" feature into XGBoost

- **Tunable threshold panel**: business adjusts recall vs\. false\-positive trade\-off in real time

**Mid\-term \(1\-2 months\)**

- **Dedicated water\-level YOLO**: pixel\-precise water\-line detection

- **Standalone video\-clarity detector**: a 5th call dedicated to "can this be assessed?"

- **Cross\-store trend dashboard**: A / B / C / D distribution over time

---

## Ready for production today

📲 Integrate into store dashboard
Daily feedback to store managers

🚨 Regional manager alert list
\-70% per\-video review time

📈 Continuous evaluation
Replay human scores to monitor AI

---

# 📎 Appendix: Technical Charts

*Below: full system architecture \+ test\-set performance, for engineering hand\-off*

![Image](https://internal-api-drive-stream-sg.larkoffice.com/space/api/box/stream/download/authcode/?code=ZTEwNjgzOGI2OTYxYTA1Y2U2MjE5ZTgzMzlmYTAzNGFfYmM4MjU0MjlhZDBlODMyYTAyM2NlNTY3YTkwNTgwNWFfSUQ6NzY0MTA3MzA4NzA5MzY0MDkyNF8xNzgxNDkxNTI3OjE3ODE1Nzc5MjdfVjM)

![Image](https://internal-api-drive-stream-sg.larkoffice.com/space/api/box/stream/download/authcode/?code=YTMwMzRhNTA4NTI3MWY1ODc4NmMxNmQ0ODk5ZTYwMThfOTc1OWUwMjc2ZTQ0ZTVkMWQ2MDE4OTJiNzZlMTgwMmVfSUQ6NzY0MTA3MzExNTQ3MjE1NDMzMl8xNzgxNDkxNTI3OjE3ODE1Nzc5MjdfVjM)

![Image](https://internal-api-drive-stream-sg.larkoffice.com/space/api/box/stream/download/authcode/?code=MDZlNThkOGRmYzEzZTMzZjhhYzc0MDJhMmQ5NGRhZjVfMTUxYTg3MmY3Njg3MDU1OWExNTVlNTNjNDIzZDQyYmNfSUQ6NzY0MTA3MzEzNTE2MDM5NzUzOF8xNzgxNDkxNTI3OjE3ODE1Nzc5MjdfVjM)

![Image](https://internal-api-drive-stream-sg.larkoffice.com/space/api/box/stream/download/authcode/?code=OWE2MTUyYjJjZTUxMDMyZWI2MGEwODU3YzNkZjUxNDNfZjVkY2JkMTZhZjJmY2QxOGFjMTViY2RjNmM3YWExMTFfSUQ6NzY0MTA3MzE1NzQxNTM1ODE3NF8xNzgxNDkxNTI3OjE3ODE1Nzc5MjdfVjM)

> （注：このコンテンツは AI によって生成されています。内容をご確認のうえご利用ください。）