33800 Docs

← Retour

Proposition : Analyse IA NeedFinder

Date : 21/12/2025 17:39 Status : IMPLEMENTEE Projet : NeedFinder Phase : 2 - Analyse IA

Contexte

Le collector NeedFinder fonctionne et accumule des posts (111 actuellement). Les posts sont stockés avec is_processed = false. Il faut maintenant implémenter l'extraction des besoins avec Ollama.

Architecture proposée

Posts (is_processed=false)
         │
         ▼
┌─────────────────────┐
│   Analyzer Worker   │
│  (src/analyzers/)   │
└─────────────────────┘
         │
         ▼
┌─────────────────────┐
│   Ollama (win11)    │
│   mistral:latest    │
└─────────────────────┘
         │
         ▼
┌─────────────────────┐
│  Insert into DB:    │
│  - needs            │
│  - mentions         │
│  - posts.is_processed=true │
└─────────────────────┘

Fichiers à créer

1. src/analyzers/need-extractor.ts

import { sql } from '../db/client';

const OLLAMA_URL = process.env.OLLAMA_URL || 'http://192.168.1.30:11434';
const MODEL = 'mistral:latest';

interface ExtractedNeed {
  hasNeed: boolean;
  needStatement?: string;
  needType?: 'wish' | 'frustration' | 'feature_request' | 'question';
  category?: string;
  confidence?: number;
}

async function callOllama(prompt: string): Promise<string> {
  const response = await fetch(`${OLLAMA_URL}/api/generate`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: MODEL,
      prompt,
      stream: false,
      options: { temperature: 0.3 }
    })
  });

  const data = await response.json();
  return data.response;
}

export async function extractNeed(post: { title: string; content: string }): Promise<ExtractedNeed> {
  const prompt = `Analyse ce post et identifie si un besoin non satisfait est exprimé.

Post:
Titre: ${post.title}
Contenu: ${post.content?.slice(0, 1000) || '(vide)'}

Réponds UNIQUEMENT en JSON valide, sans markdown:
{
  "hasNeed": true/false,
  "needStatement": "Description claire du besoin en une phrase (français)",
  "needType": "wish|frustration|feature_request|question",
  "category": "productivity|developer_tools|business|health|finance|education|entertainment|automation|other",
  "confidence": 0.0-1.0
}

Si aucun besoin clair n'est exprimé, réponds: {"hasNeed": false}`;

  try {
    const response = await callOllama(prompt);
    // Nettoyer et parser le JSON
    const jsonMatch = response.match(/\{[\s\S]*\}/);
    if (jsonMatch) {
      return JSON.parse(jsonMatch[0]);
    }
    return { hasNeed: false };
  } catch (error) {
    console.error('Ollama extraction error:', error);
    return { hasNeed: false };
  }
}

export async function processUnanalyzedPosts(limit: number = 20): Promise<{ processed: number; needsFound: number }> {
  await sql`SET search_path TO needfinder`;

  // Récupérer les posts non traités
  const posts = await sql`
    SELECT id, title, content
    FROM posts
    WHERE is_processed = false
    ORDER BY posted_at DESC
    LIMIT ${limit}
  `;

  console.log(`🔬 Analyzing ${posts.length} posts with Ollama...`);

  let processed = 0;
  let needsFound = 0;

  for (const post of posts) {
    try {
      const result = await extractNeed(post);

      if (result.hasNeed && result.needStatement) {
        // Chercher un besoin similaire existant
        const existing = await sql`
          SELECT id FROM needs
          WHERE LOWER(need_statement) = LOWER(${result.needStatement})
          LIMIT 1
        `;

        let needId: string;

        if (existing.length > 0) {
          // Incrémenter le compteur du besoin existant
          needId = existing[0].id;
          await sql`
            UPDATE needs
            SET mention_count = mention_count + 1,
                last_seen_at = NOW()
            WHERE id = ${needId}
          `;
        } else {
          // Créer un nouveau besoin
          const inserted = await sql`
            INSERT INTO needs (need_statement, category, need_type, opportunity_score)
            VALUES (${result.needStatement}, ${result.category || 'other'}, ${result.needType || 'question'}, ${Math.round((result.confidence || 0.5) * 100)})
            RETURNING id
          `;
          needId = inserted[0].id;
          needsFound++;
        }

        // Créer la mention
        await sql`
          INSERT INTO mentions (need_id, post_id, extracted_text, confidence)
          VALUES (${needId}, ${post.id}, ${result.needStatement}, ${result.confidence || 0.5})
          ON CONFLICT (need_id, post_id) DO NOTHING
        `;
      }

      // Marquer comme traité
      await sql`UPDATE posts SET is_processed = true WHERE id = ${post.id}`;
      processed++;

      // Rate limit Ollama
      await new Promise(r => setTimeout(r, 500));

    } catch (error) {
      console.error(`Error processing post ${post.id}:`, error);
    }
  }

  console.log(`✅ Analyzed ${processed} posts, found ${needsFound} new needs`);
  return { processed, needsFound };
}

2. Modifications src/index.ts

Ajouter les endpoints:

import { processUnanalyzedPosts } from './analyzers/need-extractor';

// ...existing code...

// Trigger analysis
.post('/api/analyze', async ({ query }) => {
  const limit = Math.min(parseInt(query.limit || '20'), 50);
  console.log('🔬 Manual analysis triggered via API');

  const result = await processUnanalyzedPosts(limit);
  return { status: 'completed', ...result };
})

// List needs
.get('/api/needs', async ({ query }) => {
  const limit = Math.min(parseInt(query.limit || '20'), 100);

  await sql`SET search_path TO needfinder`;

  const needs = await sql`
    SELECT n.*,
           (SELECT COUNT(*) FROM mentions m WHERE m.need_id = n.id) as mention_count_actual
    FROM needs n
    ORDER BY n.opportunity_score DESC, n.last_seen_at DESC
    LIMIT ${limit}
  `;

  return { needs, count: needs.length };
})

// Get need details with mentions
.get('/api/needs/:id', async ({ params }) => {
  await sql`SET search_path TO needfinder`;

  const need = await sql`SELECT * FROM needs WHERE id = ${params.id}`;
  const mentions = await sql`
    SELECT m.*, p.title, p.external_url, p.subreddit, s.name as source_name
    FROM mentions m
    JOIN posts p ON p.id = m.post_id
    JOIN sources s ON s.id = p.source_id
    WHERE m.need_id = ${params.id}
    ORDER BY m.created_at DESC
  `;

  return { need: need[0], mentions };
})

3. Modification src/cron/scheduler.ts

Ajouter l'analyse après la collecte:

import { processUnanalyzedPosts } from '../analyzers/need-extractor';

async function runCollection() {
  // ... existing collection code ...

  // After collection, run analysis
  console.log('🔬 Starting need extraction...');
  try {
    const analysisResult = await processUnanalyzedPosts(30);
    console.log(`   Analysis: ${analysisResult.processed} processed, ${analysisResult.needsFound} new needs`);
  } catch (error) {
    console.error('Analysis error:', error);
  }
}

Variables d'environnement

OLLAMA_URL=http://192.168.1.30:11434
OLLAMA_MODEL=mistral:latest

Workflow d'exécution

  1. Cron (toutes les 6h) :

    • Collecte Reddit + HN
    • Analyse des 30 derniers posts non traités

  2. API manuelle :

    • POST /api/analyze?limit=20 - Trigger analyse
    • GET /api/needs - Liste des besoins détectés
    • GET /api/needs/:id - Détail avec mentions

Déploiement

  1. Copier les fichiers sur dev-portainer
  2. Rebuild le container
  3. Tester avec quelques posts
  4. Observer les résultats

Test validation

# Trigger analyse
curl -X POST http://192.168.1.51:5200/api/analyze?limit=5

# Voir les besoins extraits
curl http://192.168.1.51:5200/api/needs

# Stats
curl http://192.168.1.51:5200/api/stats

Validation requise