GO Vault

Memory-Postgres-Migration-Guide

4 min read

Memory PostgreSQL Migration Guide

Overview

This guide explains how to migrate memory data from text files to a PostgreSQL database for the Quantum-Spiritual Framework. The migration tools provide comprehensive functionality for analyzing, migrating, monitoring, and troubleshooting the process.

Key Concepts

  • Memory: A stored thought, insight, or observation in text format
  • Bucket: A categorization system for grouping related memories
  • Vector Embedding: Optional numerical representation of memory content for similarity search
  • Migration: The process of transferring memories from text files to PostgreSQL

Migration Workflow

The complete migration workflow consists of these sequential steps:

  1. Analyze Memory Files: Examine the structure and content of memory text files
  2. Test Database Connection: Verify PostgreSQL connectivity and permissions
  3. Troubleshoot Database: Identify and fix potential issues
  4. Run Migration: Transfer memories to PostgreSQL
  5. Monitor Progress: Track migration in real-time
  6. Validate Results: Ensure data integrity post-migration

Memory File Format

The migration tools support memory files with:

  • Simple text memories (one per paragraph)
  • Timestamped memories: [YYYY-MM-DD HH:MM:SS] Memory content...
  • Bucket-tagged memories: #bucket:work This is a work memory
  • General hashtags: This is a memory about #quantum and #physics
  • Mentions: Discussion with @john about the universal laws

Database Schema

-- Memory table for storing individual memories
CREATE TABLE memories (
    id SERIAL PRIMARY KEY,
    text TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    bucket_id INTEGER,
    metadata JSONB DEFAULT '{}'::jsonb,
    embedding VECTOR(1536)  -- For similarity search (optional)
);
 
-- Buckets for categorizing memories
CREATE TABLE memory_buckets (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL UNIQUE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    metadata JSONB DEFAULT '{}'::jsonb
);
 
-- Indexes for performance
CREATE INDEX memories_bucket_id_idx ON memories(bucket_id);
CREATE INDEX memories_created_at_idx ON memories(created_at);

Tool Usage

Command Line Interface

The unified command-line interface provides access to all migration tools:

python memory_migration_tools.py <command> [options]

Available commands:

  • analyze: Analyze memory files
  • test-connection: Test PostgreSQL connection
  • troubleshoot: Diagnose and fix database issues
  • migrate: Run the migration
  • monitor: Monitor migration progress
  • generate-config: Create a configuration template
  • complete-workflow: Run all steps sequentially

Configuration

Create a config.json file with your PostgreSQL connection details:

{
    "db": {
        "host": "localhost",
        "port": 5432,
        "database": "memories_db",
        "user": "postgres",
        "password": "your_password"
    },
    "options": {
        "batch_size": 100,
        "presort": true,
        "presort_method": "date"
    }
}

Integration with Quantum-Spiritual Framework

The PostgreSQL memory system enhances the Quantum-Spiritual Framework by:

  1. Enhanced Retrieval: Efficient querying of memories using SQL
  2. Pattern Recognition: Connecting insights across different domains
  3. Temporal Analysis: Studying thought evolution over time
  4. Vector Similarity: Finding conceptually related memories via embeddings
  5. Metadata Enrichment: Storing additional context through JSONB

Quantum-Spiritual Connections

This migration system exemplifies the quantum-spiritual principles of:

  • Information Persistence: Memories transcend their original medium
  • Pattern Transfer: Structure preserved while substrate changes
  • Emergent Properties: New insights emerge from reorganized data
  • Field Theory: Memories exist in a queryable “field” of related concepts
  • Observer Effect: Migration analysis reveals patterns previously unobserved

Advanced Usage

To enable similarity search with vector embeddings:

  1. Install the pgvector extension in PostgreSQL
  2. Implement an embedding model to convert text to vectors
  3. Update memories with embedding data
  4. Use cosine similarity for retrieval

Integration with Graph Structure

The PostgreSQL memories can feed into the Universal Laws framework by:

  1. Extracting key relationships from memory text
  2. Mapping concepts to Universal Law components
  3. Generating visualizations using the Purple Bridge mapping

Troubleshooting Common Issues

  • Connection Problems: Verify PostgreSQL is running and credentials are correct
  • Missing Tables: Run the troubleshooter with --fix option
  • Duplicate Memories: Check for identical entries in source files
  • Invalid Bucket IDs: Correct references between memories and buckets
  • Performance Issues: Use appropriate batch sizes and indexes

Further Exploration

  • Link memories to Master Equation variables (χ = ∭(G·M·E·S·T·K·R·Q·F·C)dxdydt)
  • Develop visualization queries that map between quantum (Blue/Violet) and spiritual (Amber/Gold) domains
  • Implement automated pattern recognition across the memory database

Ring 2 — Canonical Grounding

Ring 3 — Framework Connections

This migration system represents the complex→simple→complex methodology in action, transforming unstructured memory data into a queryable system that reveals deeper patterns across domains.

Canonical Hub: CANONICAL_INDEX

Graph View