Monitor Your Model

Deployment isn’t the end—it’s the beginning of the ML lifecycle. This section covers monitoring, feedback collection, and continuous improvement.

The Feedback Loop

graph LR
    A[Deploy Model] --> B[Collect Predictions]
    B --> C[Monitor Performance]
    C --> D[Gather Feedback]
    D --> E[Identify Issues]
    E --> F[Improve Dataset]
    F --> G[Retrain Model]
    G --> A

Monitoring Dashboard

Using the Web Platform

Navigate to Models > Your Model > Analytics to view:

Prediction Volume
- Requests per hour/day/week
- Peak usage times
- Geographic distribution
Performance Metrics
- Average latency
- P95/P99 latency
- Error rate
Prediction Distribution
- Which classes are predicted most
- Confidence score distribution
- Low-confidence predictions

Monitoring Dashboard

Logging Predictions

Enable Prediction Logging

from seeme import Client

client = Client()

# Enable logging for a model
client.update_model(
    model_id=model.id,
    config={
        "log_predictions": True,
        "log_inputs": True,  # Store input images
        "log_retention_days": 30
    }
)

Query Logged Predictions

# Get recent predictions
predictions = client.get_predictions(
    model_id=model.id,
    limit=100,
    start_date="2024-01-01",
    end_date="2024-01-31"
)

for pred in predictions:
    print(f"{pred.created_at}: {pred.label} ({pred.confidence:.2%})")

Setting Up Alerts

Alert Types

Alert Type	Trigger	Example
Error Rate	Errors exceed threshold	> 5% errors
Latency	Response time too high	P99 > 500ms
Confidence	Low confidence predictions	Avg < 70%
Volume	Unusual request patterns	10x normal

Configure Alerts

# Create an alert
alert = client.create_alert(
    model_id=model.id,
    name="High Error Rate",
    metric="error_rate",
    threshold=0.05,  # 5%
    comparison="greater_than",
    window_minutes=15,
    notification={
        "type": "slack",
        "webhook_url": "https://hooks.slack.com/..."
    }
)

# Create latency alert
client.create_alert(
    model_id=model.id,
    name="High Latency",
    metric="p99_latency_ms",
    threshold=500,
    comparison="greater_than",
    window_minutes=5,
    notification={
        "type": "email",
        "recipients": ["team@example.com"]
    }
)

Collecting Feedback

User feedback improves your model over time. SeeMe.ai provides multiple ways to collect feedback.

Feedback API

# Log correct/incorrect feedback
client.log_feedback(
    prediction_id=prediction.id,
    feedback_type="correction",
    correct_label="cat",  # What the label should have been
    notes="User corrected: model said dog"
)

# Log thumbs up/down
client.log_feedback(
    prediction_id=prediction.id,
    feedback_type="rating",
    rating=1  # 1 = positive, -1 = negative
)

Feedback Widget

Embed a feedback widget in your application:

<!-- Include SeeMe feedback widget -->
<script src="https://cdn.seeme.ai/feedback-widget.js"></script>

<seeme-feedback
  model-id="your-model-id"
  prediction-id="prediction-id"
  api-key="your-api-key">
</seeme-feedback>

Review Feedback

# Get feedback for review
feedback_items = client.get_feedback(
    model_id=model.id,
    feedback_type="correction",
    limit=100
)

# Review and approve corrections
for item in feedback_items:
    print(f"Prediction: {item.predicted_label}")
    print(f"Correction: {item.correct_label}")
    print(f"Image: {item.input_url}")

    # Add approved corrections to training data
    if input("Approve? (y/n): ") == "y":
        client.approve_feedback(item.id)

Data Engine: Automated Feedback Loop

The Data Engine automatically identifies predictions that need review:

Configure Data Engine

# Enable Data Engine
client.update_model(
    model_id=model.id,
    data_engine={
        "enabled": True,
        "low_confidence_threshold": 0.7,  # Flag predictions below 70%
        "edge_case_detection": True,
        "auto_queue_for_review": True
    }
)

Review Queue

Low-confidence and edge-case predictions are automatically queued for human review:

Go to Models > Your Model > Review Queue
See predictions flagged for review
Correct labels as needed
Approved corrections are added to your dataset

Review Queue

Performance Analytics

Track Model Drift

Monitor whether model performance degrades over time:

# Get weekly performance metrics
metrics = client.get_performance_metrics(
    model_id=model.id,
    period="week",
    lookback_weeks=12
)

for week in metrics:
    print(f"{week.period}: Accuracy={week.accuracy:.2%}, "
          f"Avg Confidence={week.avg_confidence:.2%}")

Compare Model Versions

# Compare two model versions
comparison = client.compare_models(
    model_id=model.id,
    version_a=old_version.id,
    version_b=new_version.id,
    test_dataset_id=test_dataset.id
)

print(f"Version A accuracy: {comparison.version_a.accuracy:.2%}")
print(f"Version B accuracy: {comparison.version_b.accuracy:.2%}")
print(f"Improvement: {comparison.delta:.2%}")

Retraining Triggers

Set up automatic retraining when conditions are met:

# Configure auto-retrain
client.update_model(
    model_id=model.id,
    auto_retrain={
        "enabled": True,
        "triggers": [
            {
                "type": "accuracy_drop",
                "threshold": 0.05,  # Retrain if accuracy drops 5%
                "window_days": 7
            },
            {
                "type": "new_data",
                "min_items": 100  # Retrain when 100 new items added
            },
            {
                "type": "schedule",
                "cron": "0 2 * * 0"  # Weekly on Sunday at 2am
            }
        ]
    }
)

Inference Logging Best Practices

What to Log

Data	Why	Retention
Input data	Debug issues, retrain	30-90 days
Predictions	Analytics, feedback	90-365 days
Latency	Performance monitoring	30 days
Errors	Debugging	30 days
User feedback	Model improvement	Indefinite

Privacy Considerations

⚠️

Data Privacy: If logging input images, ensure compliance with GDPR and other regulations. Consider:

Anonymizing or blurring faces
Getting user consent
Setting appropriate retention periods
Enabling data deletion requests

# Configure privacy settings
client.update_model(
    model_id=model.id,
    privacy={
        "blur_faces": True,
        "anonymize_pii": True,
        "retention_days": 30,
        "user_deletion_enabled": True
    }
)

Monitoring Checklist

Ensure your monitoring covers:

Prediction volume and patterns
Error rates and types
Latency (avg, P50, P95, P99)
Confidence distribution
Model drift detection
Feedback collection
Alert configuration
Retraining triggers

Complete Lifecycle Summary

You’ve now completed the full AI lifecycle:

graph TD
    A[1. Prepare Data] --> B[2. Train Model]
    B --> C[3. Optimize]
    C --> D[4. Deploy]
    D --> E[5. Monitor]
    E -->|Feedback| A
    E -->|Auto-retrain| B

What’s Next?

Automate with Workflows Try Object Detection Team Collaboration

Video Script Outline

Hook (0:00-0:10) “Your model is live. But how do you know it’s working? Let me show you.”

What You’ll Learn (0:10-0:30)

Monitor model performance in production
Set up alerts for issues
Collect user feedback
Create a continuous improvement loop

Demo Steps (0:30-9:00)

Show monitoring dashboard (0:30)
Explain key metrics (2:00)
Configure alerts (3:30)
Show feedback collection (5:00)
Review queue walkthrough (6:30)
Set up auto-retrain (8:00)

Call to Action (9:00-10:00) “Monitoring turns your model from a one-time project into a continuously improving system. Check out our Workflows guide to see how to chain multiple models together.”

Deploy Your Model