NeedFinder - Architecture Technique

Date : 20/12/2025 Status : Draft Projet : Détection de besoins non satisfaits et opportunités de marché avec IA

1. Concept

Problème résolu

Trouver des idées de produits/services AVANT qu'ils n'existent en détectant :

• Les besoins exprimés qui n'ont pas encore de solution
• Les frustrations récurrentes
• Les demandes de fonctionnalités non satisfaites
• Les tendances émergentes avant qu'elles ne deviennent mainstream

Sources de besoins "avant-garde"

2. Vue d'ensemble

┌─────────────────────────────────────────────────────────────────────┐
│                         DASHBOARD                                    │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐  │
│  │ Trending │ │  Needs   │ │  Gaps    │ │ Research │ │  Alerts  │  │
│  │  Needs   │ │  Search  │ │ Analysis │ │  Papers  │ │  Setup   │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘  │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         COLLECTORS                                   │
│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │
│  │ Reddit │ │   HN   │ │Twitter │ │ GitHub │ │  SO    │ │ Arxiv  │ │
│  └────────┘ └────────┘ └────────┘ └────────┘ └────────┘ └────────┘ │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        AI ANALYSIS                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │
│  │    Ollama    │  │   pgvector   │  │  Clustering  │              │
│  │   Classify   │  │  Embeddings  │  │   Similar    │              │
│  │   Extract    │  │   Semantic   │  │    Needs     │              │
│  └──────────────┘  └──────────────┘  └──────────────┘              │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      OUTPUT                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │
│  │   Trending   │  │  Gap Report  │  │   Telegram   │              │
│  │    Needs     │  │  (Analysis)  │  │    Alerts    │              │
│  └──────────────┘  └──────────────┘  └──────────────┘              │
└─────────────────────────────────────────────────────────────────────┘

3. Stack Technique

Backend

Collectors

IA

Frontend

4. Patterns de détection

Phrases clés (Need Signals)

const NEED_PATTERNS = {
    direct_wish: [
        "I wish there was",
        "I wish I could",
        "Why isn't there",
        "Why doesn't exist",
        "Someone should build",
        "Would pay for",
        "Looking for a tool",
        "Is there any app that",
        "Does anyone know of",
    ],

    frustration: [
        "I hate when",
        "So frustrating that",
        "Can't believe there's no",
        "Tired of",
        "Fed up with",
        "Why is it so hard to",
        "Wasted hours trying to",
    ],

    feature_request: [
        "Would be great if",
        "Feature request:",
        "Should have",
        "Missing feature",
        "Needs to support",
        "Please add",
    ],

    question_need: [
        "How do you",
        "What do you use for",
        "Best way to",
        "How to automate",
        "Alternative to",
    ],

    validation: [
        "Would you use",
        "Would you pay for",
        "Is there a market for",
        "Validating idea",
    ]
};

Scoring d'opportunité

interface OpportunityScore {
    // Volume
    mentionCount: number;           // Nombre de mentions
    uniqueSources: number;          // Diversité des sources
    growthRate: number;             // Tendance (croissance mentions)

    // Intensité
    sentimentIntensity: number;     // Force de la frustration/besoin
    urgencyWords: number;           // "urgent", "asap", "critical"
    willingnessToPay: boolean;      // Mention de paiement

    // Marché
    targetAudience: string[];       // Segments identifiés
    existingSolutions: number;      // Concurrents trouvés
    solutionGap: number;            // Écart avec solutions existantes

    // Score final
    opportunityScore: number;       // 0-100
}

5. Schéma Base de Données

-- ============================================
-- SOURCES
-- ============================================

CREATE TABLE sources (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name VARCHAR(100) NOT NULL, -- reddit, hackernews, twitter, etc
    source_type VARCHAR(50), -- api, scrape, rss
    config JSONB, -- API keys, rate limits, etc
    is_active BOOLEAN DEFAULT true,
    last_collected_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- ============================================
-- RAW DATA (Posts collectés)
-- ============================================

CREATE TABLE raw_posts (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    source_id UUID REFERENCES sources(id),

    -- Identifiant unique par source
    external_id VARCHAR(200),
    external_url TEXT,

    -- Contenu
    title TEXT,
    content TEXT,
    author VARCHAR(200),

    -- Métadonnées source
    subreddit VARCHAR(100), -- Reddit
    score INTEGER, -- Upvotes
    comment_count INTEGER,

    -- Dates
    posted_at TIMESTAMPTZ,
    collected_at TIMESTAMPTZ DEFAULT NOW(),

    -- Embedding
    embedding vector(384),

    -- Processing
    is_processed BOOLEAN DEFAULT false,

    UNIQUE(source_id, external_id)
);

CREATE INDEX idx_posts_source ON raw_posts(source_id, posted_at DESC);
CREATE INDEX idx_posts_embedding ON raw_posts USING ivfflat (embedding vector_cosine_ops);

-- ============================================
-- NEEDS DÉTECTÉS
-- ============================================

CREATE TABLE detected_needs (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),

    -- Le besoin extrait
    need_statement TEXT NOT NULL, -- "Un outil pour automatiser X"
    need_category VARCHAR(100), -- productivity, developer_tools, health, etc
    need_type VARCHAR(50), -- wish, frustration, feature_request, question

    -- Embedding pour clustering
    embedding vector(384),

    -- Scoring
    mention_count INTEGER DEFAULT 1,
    unique_sources INTEGER DEFAULT 1,
    avg_sentiment DECIMAL(3,2), -- -1 à +1
    urgency_score INTEGER, -- 1-10
    willingness_to_pay BOOLEAN DEFAULT false,
    opportunity_score INTEGER, -- 0-100

    -- Audience
    target_segments TEXT[],
    estimated_market_size VARCHAR(50), -- small, medium, large

    -- Competition
    existing_solutions TEXT[],
    solution_gap_score INTEGER, -- 1-10

    -- Timeline
    first_seen_at TIMESTAMPTZ DEFAULT NOW(),
    last_seen_at TIMESTAMPTZ DEFAULT NOW(),
    trending BOOLEAN DEFAULT false,

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_needs_score ON detected_needs(opportunity_score DESC);
CREATE INDEX idx_needs_category ON detected_needs(need_category);
CREATE INDEX idx_needs_trending ON detected_needs(trending, opportunity_score DESC);

-- ============================================
-- MENTIONS (Lien posts → needs)
-- ============================================

CREATE TABLE need_mentions (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    need_id UUID REFERENCES detected_needs(id) ON DELETE CASCADE,
    post_id UUID REFERENCES raw_posts(id) ON DELETE CASCADE,

    -- Extraction
    extracted_text TEXT, -- La phrase exacte
    confidence DECIMAL(3,2), -- Confiance IA

    created_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(need_id, post_id)
);

-- ============================================
-- CLUSTERS DE BESOINS SIMILAIRES
-- ============================================

CREATE TABLE need_clusters (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name VARCHAR(300), -- Nom généré
    description TEXT,

    -- Stats
    need_count INTEGER DEFAULT 0,
    total_mentions INTEGER DEFAULT 0,
    avg_opportunity_score DECIMAL(5,2),

    -- Représentant
    representative_need_id UUID REFERENCES detected_needs(id),

    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE cluster_members (
    cluster_id UUID REFERENCES need_clusters(id) ON DELETE CASCADE,
    need_id UUID REFERENCES detected_needs(id) ON DELETE CASCADE,
    similarity_score DECIMAL(3,2),
    PRIMARY KEY (cluster_id, need_id)
);

-- ============================================
-- SOLUTIONS EXISTANTES
-- ============================================

CREATE TABLE existing_solutions (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),

    name VARCHAR(200) NOT NULL,
    url TEXT,
    description TEXT,

    -- Catégorie
    categories TEXT[],

    -- Pricing
    pricing_model VARCHAR(50), -- free, freemium, paid, enterprise
    price_range VARCHAR(50),

    -- Ratings
    rating DECIMAL(2,1),
    user_count VARCHAR(50), -- estimation

    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE need_solutions (
    need_id UUID REFERENCES detected_needs(id) ON DELETE CASCADE,
    solution_id UUID REFERENCES existing_solutions(id) ON DELETE CASCADE,
    match_score DECIMAL(3,2), -- Correspond à quel point
    gaps TEXT[], -- Ce que la solution ne fait pas
    PRIMARY KEY (need_id, solution_id)
);

-- ============================================
-- RECHERCHE ACADÉMIQUE
-- ============================================

CREATE TABLE research_papers (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),

    title TEXT NOT NULL,
    abstract TEXT,
    authors TEXT[],
    arxiv_id VARCHAR(50),
    url TEXT,

    -- Classification
    categories TEXT[],
    keywords TEXT[],

    -- Applications potentielles
    potential_applications TEXT,
    commercial_viability VARCHAR(50), -- low, medium, high

    -- Embedding
    embedding vector(384),

    published_at DATE,
    collected_at TIMESTAMPTZ DEFAULT NOW()
);

-- ============================================
-- ALERTES UTILISATEUR
-- ============================================

CREATE TABLE alerts (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    user_id UUID REFERENCES auth.users(id) ON DELETE CASCADE,

    name VARCHAR(200),
    alert_type VARCHAR(50), -- keyword, category, threshold

    -- Conditions
    keywords TEXT[],
    categories TEXT[],
    min_opportunity_score INTEGER,
    min_mentions INTEGER,

    -- Notification
    notify_telegram BOOLEAN DEFAULT true,
    notify_email BOOLEAN DEFAULT false,

    is_active BOOLEAN DEFAULT true,
    last_triggered_at TIMESTAMPTZ,

    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- ============================================
-- HISTORIQUE COLLECTION
-- ============================================

CREATE TABLE collection_runs (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    source_id UUID REFERENCES sources(id),
    status VARCHAR(20), -- running, completed, failed
    posts_collected INTEGER DEFAULT 0,
    needs_extracted INTEGER DEFAULT 0,
    error_message TEXT,
    started_at TIMESTAMPTZ DEFAULT NOW(),
    completed_at TIMESTAMPTZ
);

6. Collectors

Reddit Collector

// /workers/collectors/reddit.ts
import Snoowrap from 'snoowrap';

const reddit = new Snoowrap({
    userAgent: 'NeedFinder/1.0',
    clientId: process.env.REDDIT_CLIENT_ID,
    clientSecret: process.env.REDDIT_SECRET,
    refreshToken: process.env.REDDIT_REFRESH_TOKEN
});

const TARGET_SUBREDDITS = [
    'SomebodyMakeThis',      // Demandes directes de produits
    'AppIdeas',              // Idées d'apps
    'SideProject',           // Projets side
    'Entrepreneur',          // Besoins business
    'startups',              // Startups
    'indiehackers',          // Indie hackers
    'webdev',                // Dev web
    'selfhosted',            // Self-hosted
    'productivity',          // Productivité
    'automation',            // Automatisation
    'artificial',            // IA
    'MachineLearning',       // ML
];

const NEED_SUBREDDITS = [
    'IsThereAnAppForThat',   // Recherche d'apps
    'AskEngineers',          // Questions techniques
    'NoStupidQuestions',     // Questions diverses
];

async function collectReddit() {
    for (const subreddit of TARGET_SUBREDDITS) {
        const posts = await reddit.getSubreddit(subreddit).getNew({ limit: 100 });

        for (const post of posts) {
            // Vérifier si match un pattern de besoin
            if (matchesNeedPattern(post.title + ' ' + post.selftext)) {
                await savePost({
                    source: 'reddit',
                    externalId: post.id,
                    externalUrl: `https://reddit.com${post.permalink}`,
                    title: post.title,
                    content: post.selftext,
                    author: post.author.name,
                    subreddit: subreddit,
                    score: post.score,
                    commentCount: post.num_comments,
                    postedAt: new Date(post.created_utc * 1000)
                });
            }
        }
    }
}

function matchesNeedPattern(text: string): boolean {
    const patterns = [
        /i wish there was/i,
        /is there (a|an|any) (app|tool|service|software)/i,
        /looking for (a|an) (tool|app|way to)/i,
        /would pay for/i,
        /someone should (build|make|create)/i,
        /why (isn't|doesn't|can't) there/i,
        /frustrated with/i,
        /alternative to/i,
    ];

    return patterns.some(p => p.test(text));
}

Hacker News Collector

// /workers/collectors/hackernews.ts
import { HackerNewsApi } from 'hn-api';

const KEYWORDS = [
    'Ask HN:',
    'Show HN:',
    'looking for',
    'alternative to',
    'how do you',
    'best tool for',
];

async function collectHN() {
    // Top stories
    const topIds = await fetch('https://hacker-news.firebaseio.com/v0/topstories.json')
        .then(r => r.json());

    // New stories
    const newIds = await fetch('https://hacker-news.firebaseio.com/v0/newstories.json')
        .then(r => r.json());

    // Ask HN
    const askIds = await fetch('https://hacker-news.firebaseio.com/v0/askstories.json')
        .then(r => r.json());

    for (const id of [...topIds.slice(0, 50), ...askIds.slice(0, 100)]) {
        const item = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`)
            .then(r => r.json());

        if (item.type === 'story' && matchesNeedPattern(item.title)) {
            await savePost({
                source: 'hackernews',
                externalId: String(item.id),
                externalUrl: `https://news.ycombinator.com/item?id=${item.id}`,
                title: item.title,
                content: item.text || '',
                author: item.by,
                score: item.score,
                commentCount: item.descendants,
                postedAt: new Date(item.time * 1000)
            });
        }
    }
}

Twitter/X Collector

// /workers/collectors/twitter.ts
// Note: Nécessite API payante ou scraping

const SEARCH_QUERIES = [
    '"I wish there was an app"',
    '"looking for a tool"',
    '"is there an app for"',
    '"why doesn\'t exist"',
    '"would pay for"',
    '"frustrated with"',
    'startup idea',
];

async function collectTwitter() {
    for (const query of SEARCH_QUERIES) {
        const tweets = await searchTweets(query, { maxResults: 100 });

        for (const tweet of tweets) {
            await savePost({
                source: 'twitter',
                externalId: tweet.id,
                externalUrl: `https://twitter.com/i/status/${tweet.id}`,
                title: '',
                content: tweet.text,
                author: tweet.author_id,
                score: tweet.public_metrics.like_count,
                postedAt: new Date(tweet.created_at)
            });
        }
    }
}

Arxiv Collector (Recherche)

// /workers/collectors/arxiv.ts
import Parser from 'rss-parser';

const ARXIV_CATEGORIES = [
    'cs.AI',      // Artificial Intelligence
    'cs.LG',      // Machine Learning
    'cs.CL',      // Computation and Language (NLP)
    'cs.CV',      // Computer Vision
    'cs.HC',      // Human-Computer Interaction
];

async function collectArxiv() {
    const parser = new Parser();

    for (const category of ARXIV_CATEGORIES) {
        const feed = await parser.parseURL(
            `http://export.arxiv.org/rss/${category}`
        );

        for (const item of feed.items) {
            await savePaper({
                source: 'arxiv',
                externalId: item.id,
                title: item.title,
                abstract: item.summary,
                authors: extractAuthors(item),
                categories: [category],
                url: item.link,
                publishedAt: new Date(item.pubDate)
            });
        }
    }
}

7. Analyse IA

Extracteur de besoins

// /workers/analyzers/need-extractor.ts

async function extractNeed(post: RawPost): Promise<ExtractedNeed | null> {
    const prompt = `
Analyse ce post et extrait le besoin exprimé s'il y en a un.

Post:
Titre: ${post.title}
Contenu: ${post.content}

Si un besoin est exprimé, réponds en JSON:
{
    "hasNeed": true,
    "needStatement": "Description claire du besoin en une phrase",
    "needType": "wish|frustration|feature_request|question",
    "urgency": 1-10,
    "willingnessToPay": true/false,
    "targetAudience": ["segment1", "segment2"],
    "category": "productivity|developer_tools|health|finance|education|entertainment|other",
    "existingSolutionsMentioned": ["solution1", "solution2"]
}

Si pas de besoin clair, réponds:
{
    "hasNeed": false
}
`;

    const response = await fetch('http://192.168.1.30:11434/api/generate', {
        method: 'POST',
        body: JSON.stringify({
            model: 'mistral:7b',
            prompt,
            options: { temperature: 0.3 }
        })
    });

    return JSON.parse(await response.text());
}

Clustering de besoins similaires

// /workers/analyzers/clustering.ts
import { kmeans } from 'ml-kmeans';

async function clusterSimilarNeeds() {
    // Récupérer tous les besoins avec embeddings
    const needs = await supabase
        .from('detected_needs')
        .select('id, need_statement, embedding')
        .not('embedding', 'is', null);

    // K-means clustering
    const embeddings = needs.data.map(n => n.embedding);
    const k = Math.min(20, Math.floor(needs.data.length / 5));

    const result = kmeans(embeddings, k, {});

    // Créer les clusters
    for (let i = 0; i < k; i++) {
        const clusterNeeds = needs.data.filter((_, idx) => result.clusters[idx] === i);

        // Générer un nom pour le cluster
        const clusterName = await generateClusterName(clusterNeeds);

        const cluster = await supabase.from('need_clusters').insert({
            name: clusterName,
            need_count: clusterNeeds.length,
            representative_need_id: clusterNeeds[0].id
        }).select().single();

        // Ajouter les membres
        for (const need of clusterNeeds) {
            await supabase.from('cluster_members').insert({
                cluster_id: cluster.data.id,
                need_id: need.id
            });
        }
    }
}

Scoring d'opportunité

// /workers/analyzers/opportunity-scorer.ts

async function scoreOpportunity(needId: string): Promise<number> {
    const need = await getNeedWithMentions(needId);

    let score = 0;

    // Volume (0-25 points)
    score += Math.min(25, need.mentionCount * 2);

    // Diversité sources (0-15 points)
    score += Math.min(15, need.uniqueSources * 5);

    // Croissance (0-15 points)
    const growth = calculateGrowthRate(need.mentions);
    score += Math.min(15, growth * 3);

    // Urgence (0-15 points)
    score += (need.urgencyScore / 10) * 15;

    // Willingness to pay (0-15 points)
    if (need.willingnessToPay) score += 15;

    // Gap vs solutions (0-15 points)
    const solutions = await findExistingSolutions(need.needStatement);
    const gapScore = 10 - Math.min(10, solutions.length);
    score += (gapScore / 10) * 15;

    return Math.round(score);
}

8. Interface

Dashboard principal

┌─────────────────────────────────────────────────────────────────────┐
│  NeedFinder                    [Alertes: 3 new]  [Sources] [⚙️]     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  🔥 TRENDING NEEDS (7 derniers jours)                               │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  #1  "Outil pour convertir Figma → code Qwik"      Score: 87   ││
│  │      📊 42 mentions  📈 +156%  💰 Willingness: Yes             ││
│  │      Sources: Reddit (23), HN (12), Twitter (7)                ││
│  │      [Voir détails] [Analyser gap]                             ││
│  │  ──────────────────────────────────────────────────────────────││
│  │  #2  "Alternative open-source à Notion + AI"       Score: 82   ││
│  │      📊 38 mentions  📈 +89%   💰 Willingness: Yes             ││
│  │  ──────────────────────────────────────────────────────────────││
│  │  #3  "Automatiser la création de tests E2E"        Score: 75   ││
│  │      📊 28 mentions  📈 +45%   💰 Willingness: Maybe           ││
│  └─────────────────────────────────────────────────────────────────┘│
│                                                                     │
│  📂 PAR CATÉGORIE                                                   │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐  │
│  │Developer │ │Producti- │ │  Health  │ │  Finance │ │Education │  │
│  │  Tools   │ │   vity   │ │          │ │          │ │          │  │
│  │  (156)   │ │  (134)   │ │   (89)   │ │   (67)   │ │   (45)   │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘  │
│                                                                     │
│  🔬 RESEARCH PAPERS (Applications potentielles)                     │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  "Zero-shot code generation with LLMs"  → Opportunité: RapidAPI││
│  │  "Real-time emotion detection from voice" → Opportunité: Health ││
│  └─────────────────────────────────────────────────────────────────┘│
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

9. MVP - Phases

Phase 1 : Collection (2 semaines)

• [ ] Collector Reddit (subreddits clés)
• [ ] Collector Hacker News
• [ ] Stockage raw posts
• [ ] Détection patterns basique

Phase 2 : Analyse IA (2 semaines)

• [ ] Extraction besoins Ollama
• [ ] Embeddings pgvector
• [ ] Clustering besoins similaires
• [ ] Scoring opportunité

Phase 3 : Dashboard (2 semaines)

• [ ] Interface trending
• [ ] Recherche sémantique
• [ ] Filtres par catégorie
• [ ] Détail need avec sources

Phase 4 : Alertes (1 semaine)

• [ ] Configuration alertes
• [ ] Notifications Telegram
• [ ] Digest email

Phase 5 : Research (optionnel)

• [ ] Collector Arxiv
• [ ] Matching papers → opportunités

10. Déploiement

services:
  needfinder-app:
    build: ./app
    ports:
      - "5100:3000"

  needfinder-api:
    build: ./api
    ports:
      - "5101:3001"

  needfinder-collectors:
    build: ./collectors
    # Cron jobs

  needfinder-analyzers:
    build: ./analyzers
    # Workers queue

URLs