diff --git a/README.md b/README.md index a6dc623..72b7d29 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,9 @@ The project is intentionally small and operationally explicit: everything local - Stores memories inside explicit projects. - Supports four categories: `requirement`, `location`, `todo`, and `note`. -- Finds memories with hybrid semantic and keyword search. -- Updates embeddings whenever memory content changes. +- Finds memories with hybrid semantic and keyword search across one to four query texts. +- Stores one default embedding per memory and optional additional embeddings for targeted discoverability. +- Updates the default embedding whenever memory content changes. - Soft-deletes items so accidental deletes are recoverable at the database level. - Exposes everything through HTTP MCP tools. - Provides an optional Blazor web UI for browsing and testing memories. @@ -102,12 +103,22 @@ All local operations are script-first. | --- | --- | | `CreateProject` | Create an explicit project. | | `ListProjects` | List projects. | -| `AddItem` | Add a memory item to an existing project. | +| `AddRequirement` | Add a requirement with required statement and context fields. | +| `AddLocation` | Add a location with required item and place fields. | +| `AddTodo` | Add a todo with required task and priority fields. | +| `AddNote` | Add a note with required subject and body fields. | | `FindItems` | Search memory items. | -| `UpdateItem` | Update an item and regenerate embeddings when needed. | +| `ListItemEmbeddings` | List default and additional embeddings for an item. | +| `AddItemEmbedding` | Add an additional discoverability embedding. | +| `UpdateItemEmbedding` | Update an additional embedding. | +| `DeleteItemEmbedding` | Delete an additional embedding. | +| `UpdateRequirement` | Update a requirement item. | +| `UpdateLocation` | Update a location item. | +| `UpdateTodo` | Update a todo item. | +| `UpdateNote` | Update a note item. | | `DeleteItem` | Soft-delete an item. | -There is no default project. `AddItem`, `UpdateItem`, and `DeleteItem` require an existing project. `FindItems` requires a project unless `searchAllProjects` is set to `true`. +There is no default project. Add, update, and delete tools require an existing project. `FindItems` requires a project unless `searchAllProjects` is set to `true`. ## Documentation diff --git a/docs/architecture.md b/docs/architecture.md index 9b70180..5421cf6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -61,9 +61,9 @@ Fields: ### Memory Item -Memory items belong to exactly one project. +Memory items belong to exactly one project. EF Core maps them with table-per-type inheritance: shared fields live in `memory_items`, and each category has a subtype table with required category-specific columns. -Fields: +Base fields: - `id` - `project_id` @@ -71,34 +71,49 @@ Fields: - `title` - `content` - `tags` -- `metadata` - `status` -- `embedding` -- `embedding_model` - `created_at` - `updated_at` - `deleted_at` -The `embedding` column is a `vector(384)` pgvector column. +### Memory Item Embedding + +Embeddings are stored separately from items so a large item can expose additional targeted discoverability text without changing its canonical description. + +Fields: + +- `id` +- `memory_item_id` +- `position` +- `label` +- `text` +- `embedding` +- `embedding_model` +- `created_at` +- `updated_at` + +The `embedding` column is a `vector(384)` pgvector column. Position `0` is the generated default embedding. Additional embeddings use position `1` and above. ## Categories -| Category | Use | -| --- | --- | -| `requirement` | Project requirements, decisions, constraints. | -| `location` | Where something is stored or placed. | -| `todo` | Action items. | -| `note` | Ideas and general memories. | +| Category | Table | Required fields | +| --- | --- | --- | +| `requirement` | `requirement_memory_items` | `statement`, `context` | +| `location` | `location_memory_items` | `item_name`, `place` | +| `todo` | `todo_memory_items` | `task`, `priority` | +| `note` | `note_memory_items` | `subject`, `body` | ## Search Search is hybrid: - PostgreSQL full-text ranking over `title` and `content` -- pgvector cosine similarity over embeddings +- pgvector cosine similarity over all embeddings linked to each item - Project and category filters - Recency as a secondary sort +`FindItems` accepts one to four query texts. Cortex embeds each query, scores every query against every item embedding, and returns each item once using its best matching query/embedding pair. + Normal search requires a project. Global search requires `searchAllProjects = true`. ## Embeddings @@ -109,17 +124,17 @@ The embedding model is: BAAI/bge-small-en-v1.5 ``` -The model produces 384-dimensional vectors. That dimension is part of the database schema, so changing models later requires a planned migration or a second embedding column. +The model produces 384-dimensional vectors. That dimension is part of the embedding table schema, so changing models later requires a planned re-embedding migration or a parallel embedding table/column strategy. Embeddings are generated for: - new items -- title changes -- content changes -- category changes +- category-specific field changes - tag changes -Metadata and status changes do not regenerate embeddings. +Additional embeddings are generated when they are added or when their text changes. Additional embedding text is only for discoverability; the item still has one canonical title/content description. + +Status changes do not regenerate embeddings. ## Deletes diff --git a/docs/development.md b/docs/development.md index 48139fa..02123a6 100644 --- a/docs/development.md +++ b/docs/development.md @@ -63,7 +63,7 @@ The current model produces 384-dimensional vectors. If the new model has a diffe - indexes - docs -Changing dimensions for existing data requires either a re-embedding migration or a new embedding column. +Changing dimensions for existing data requires either a re-embedding migration or a new embedding table/column strategy. ## Local Docker Config diff --git a/docs/mcp-tools.md b/docs/mcp-tools.md index 410f622..425873c 100644 --- a/docs/mcp-tools.md +++ b/docs/mcp-tools.md @@ -22,113 +22,190 @@ Arguments: | --- | --- | --- | --- | | `name` | string | yes | Human-readable project name. | -Result: - -```json -{ - "id": "00000000-0000-0000-0000-000000000000", - "name": "Cortex", - "slug": "cortex", - "createdAt": "2026-07-08T12:00:00Z" -} -``` - ## ListProjects Lists projects. Arguments: none. -Result: +## AddRequirement -```json -[ - { - "id": "00000000-0000-0000-0000-000000000000", - "name": "Cortex", - "slug": "cortex", - "createdAt": "2026-07-08T12:00:00Z" - } -] -``` - -## AddItem - -Adds a memory item to an existing project. - -Arguments: +Adds a requirement, decision, or constraint. | Name | Type | Required | Description | | --- | --- | --- | --- | | `project` | string | yes | Existing project name or slug. | -| `category` | string | yes | `requirement`, `location`, `todo`, or `note`. | -| `title` | string | yes | Short title. | -| `content` | string | yes | Full memory content. | +| `statement` | string | yes | Requirement, decision, or constraint statement. | +| `context` | string | yes | Context explaining why it matters. | +| `tags` | string array | no | Optional tags. | +| `status` | string | no | Optional status. Defaults to `active`. | + +## AddLocation + +Adds where a concrete thing is located. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `itemName` | string | yes | Object, file, or resource being located. | +| `place` | string | yes | Place where the item is located. | +| `details` | string | no | Extra location details. | | `tags` | string array | no | Optional tags. | -| `metadataJson` | string | no | Optional JSON object encoded as a string. | | `status` | string | no | Optional status. Defaults to `active`. | Example: ```json { - "project": "cortex", - "category": "location", - "title": "USB-C charging brick", - "content": "The USB-C charging brick is in the blue backpack.", - "tags": ["charging", "backpack"], - "metadataJson": "{\"object\":\"charging brick\",\"place\":\"blue backpack\"}" + "project": "home", + "itemName": "USB-C charging brick", + "place": "blue backpack", + "tags": ["charging", "backpack"] } ``` +## AddTodo + +Adds an action item. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `task` | string | yes | Action to complete. | +| `priority` | string | yes | Priority such as `low`, `normal`, `high`, or `urgent`. | +| `dueAt` | timestamp | no | Optional due timestamp. | +| `details` | string | no | Extra task details. | +| `tags` | string array | no | Optional tags. | +| `status` | string | no | Optional status. Defaults to `active`. | + +## AddNote + +Adds a general note or idea. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `subject` | string | yes | Note subject. | +| `body` | string | yes | Note body. | +| `tags` | string array | no | Optional tags. | +| `status` | string | no | Optional status. Defaults to `active`. | + ## FindItems Finds memories with hybrid search. -Arguments: +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `queries` | string array | yes | One to four natural-language search queries. Each query is embedded and searched separately. | +| `project` | string | required unless global | Project name or slug. | +| `searchAllProjects` | boolean | no | Set `true` for global search. | +| `category` | string | no | Optional category filter: `requirement`, `location`, `todo`, or `note`. | +| `limit` | integer | no | Result limit from 1 to 50. Defaults to 10. | + +Results are distinct items. If an item has multiple embeddings, Cortex returns the best matching embedding/query pair with `matchedQueryIndex`, `matchedEmbeddingPosition`, and `matchedEmbeddingLabel`. + +## ListItemEmbeddings + +Lists the default and additional embeddings for an item. | Name | Type | Required | Description | | --- | --- | --- | --- | -| `query` | string | yes | Natural-language search query. | -| `project` | string | required unless global | Project name or slug. | -| `searchAllProjects` | boolean | no | Set `true` for global search. | -| `category` | string | no | Optional category filter. | -| `limit` | integer | no | Result limit from 1 to 50. Defaults to 10. | +| `project` | string | yes | Existing project name or slug. | +| `itemId` | GUID | yes | Item id from a previous result. | -Example: +## AddItemEmbedding -```json -{ - "query": "where is my charging brick", - "project": "home", - "category": "location", - "limit": 5 -} -``` +Adds an additional discoverability embedding. This does not change the item's canonical title/content. -## UpdateItem +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `itemId` | GUID | yes | Item id from a previous result. | +| `text` | string | yes | Aspect-specific text to embed. | +| `label` | string | no | Short label for what the embedding targets. | -Updates an existing item. If title, content, category, or tags change, Cortex regenerates the embedding. +## UpdateItemEmbedding -Arguments: +Updates an additional embedding. The generated default embedding at position `0` cannot be edited directly. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `itemId` | GUID | yes | Item id from a previous result. | +| `embeddingId` | GUID | yes | Embedding id from `ListItemEmbeddings`. | +| `label` | string | no | Replacement label. Pass an empty string to clear. | +| `text` | string | no | Replacement embedding text. | + +## DeleteItemEmbedding + +Deletes an additional embedding. The default embedding at position `0` cannot be deleted. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `itemId` | GUID | yes | Item id from a previous result. | +| `embeddingId` | GUID | yes | Embedding id from `ListItemEmbeddings`. | + +## UpdateRequirement + +Updates an existing requirement item. | Name | Type | Required | Description | | --- | --- | --- | --- | | `project` | string | yes | Existing project name or slug. | | `id` | GUID | yes | Item id from a previous result. | -| `category` | string | no | Replacement category. | -| `title` | string | no | Replacement title. | -| `content` | string | no | Replacement content. | +| `statement` | string | no | Replacement statement. | +| `context` | string | no | Replacement context. | +| `tags` | string array | no | Replacement tag list. | +| `status` | string | no | Replacement status. | + +## UpdateLocation + +Updates an existing location item. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `id` | GUID | yes | Item id from a previous result. | +| `itemName` | string | no | Replacement item name. | +| `place` | string | no | Replacement place. | +| `details` | string | no | Replacement details. Pass an empty string to clear. | +| `tags` | string array | no | Replacement tag list. | +| `status` | string | no | Replacement status. | + +## UpdateTodo + +Updates an existing todo item. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `id` | GUID | yes | Item id from a previous result. | +| `task` | string | no | Replacement task. | +| `priority` | string | no | Replacement priority. | +| `dueAt` | timestamp | no | Replacement due timestamp. | +| `clearDueAt` | boolean | no | Set `true` to clear the due timestamp. | +| `details` | string | no | Replacement details. Pass an empty string to clear. | +| `tags` | string array | no | Replacement tag list. | +| `status` | string | no | Replacement status. | + +## UpdateNote + +Updates an existing note item. + +| Name | Type | Required | Description | +| --- | --- | --- | --- | +| `project` | string | yes | Existing project name or slug. | +| `id` | GUID | yes | Item id from a previous result. | +| `subject` | string | no | Replacement subject. | +| `body` | string | no | Replacement body. | | `tags` | string array | no | Replacement tag list. | -| `metadataJson` | string | no | Replacement JSON metadata string. | | `status` | string | no | Replacement status. | ## DeleteItem Soft-deletes an item. -Arguments: - | Name | Type | Required | Description | | --- | --- | --- | --- | | `project` | string | yes | Existing project name or slug. | diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 338872e..f608787 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -50,9 +50,9 @@ Check logs: .\scripts\logs.ps1 -Service app -Follow ``` -## AddItem Fails with Missing Project +## Add Tool Fails with Missing Project -This is expected when the project does not exist. Create the project first with `CreateProject`. +This is expected when the project does not exist. Create the project first with `CreateProject`, then use the category-specific add tool. Cortex intentionally has no default project. diff --git a/docs/web-ui.md b/docs/web-ui.md index 1d38418..bb8dcd4 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -55,11 +55,12 @@ Stop: - Create projects. - Browse project memory counts. -- Add memories in `requirement`, `location`, `todo`, and `note` categories. -- Search within one project or all projects. +- Add memories through category-specific forms for `requirement`, `location`, `todo`, and `note`. +- Search with one to four query texts within one project or all projects. - Filter recent memories by category. - Include or hide soft-deleted items. - Edit active memories. +- Add and delete additional discoverability embeddings from the edit modal. - Soft-delete active memories. ## Architecture diff --git a/src/Cortex.Api/Tools/CortexTools.cs b/src/Cortex.Api/Tools/CortexTools.cs index 3f73268..6dc983b 100644 --- a/src/Cortex.Api/Tools/CortexTools.cs +++ b/src/Cortex.Api/Tools/CortexTools.cs @@ -27,51 +27,191 @@ public static class CortexTools return memory.ListProjectsAsync(cancellationToken); } - [McpServerTool(Name = "AddItem", UseStructuredContent = true, Destructive = false)] - [Description("Add a memory item to an existing explicit project.")] - public static Task AddItem( + [McpServerTool(Name = "AddRequirement", UseStructuredContent = true, Destructive = false)] + [Description("Add a project requirement, decision, or constraint to an existing explicit project.")] + public static Task AddRequirement( ICortexMemoryService memory, [Description("Existing project name or slug. The project must already exist.")] string project, - [Description("One of: requirement, location, todo, note.")] string category, - [Description("Short item title.")] string title, - [Description("Full item content to remember.")] string content, + [Description("Required requirement, decision, or constraint statement.")] string statement, + [Description("Required context explaining why this requirement matters.")] string context, [Description("Optional tags for filtering and recall.")] string[]? tags = null, - [Description("Optional JSON object with category-specific fields.")] string? metadataJson = null, [Description("Optional item status. Defaults to active.")] string? status = null, CancellationToken cancellationToken = default) { - return memory.AddItemAsync(new AddMemoryItemRequest(project, category, title, content, tags, metadataJson, status), cancellationToken); + return memory.AddRequirementAsync(new AddRequirementRequest(project, statement, context, tags, status), cancellationToken); + } + + [McpServerTool(Name = "AddLocation", UseStructuredContent = true, Destructive = false)] + [Description("Add where a concrete thing is located in an existing explicit project.")] + public static Task AddLocation( + ICortexMemoryService memory, + [Description("Existing project name or slug. The project must already exist.")] string project, + [Description("Required name of the object, file, or resource being located.")] string itemName, + [Description("Required place where the item is located.")] string place, + [Description("Optional extra location details.")] string? details = null, + [Description("Optional tags for filtering and recall.")] string[]? tags = null, + [Description("Optional item status. Defaults to active.")] string? status = null, + CancellationToken cancellationToken = default) + { + return memory.AddLocationAsync(new AddLocationRequest(project, itemName, place, details, tags, status), cancellationToken); + } + + [McpServerTool(Name = "AddTodo", UseStructuredContent = true, Destructive = false)] + [Description("Add an action item to an existing explicit project.")] + public static Task AddTodo( + ICortexMemoryService memory, + [Description("Existing project name or slug. The project must already exist.")] string project, + [Description("Required action to complete.")] string task, + [Description("Required priority such as low, normal, high, or urgent.")] string priority, + [Description("Optional due timestamp.")] DateTimeOffset? dueAt = null, + [Description("Optional extra task details.")] string? details = null, + [Description("Optional tags for filtering and recall.")] string[]? tags = null, + [Description("Optional item status. Defaults to active.")] string? status = null, + CancellationToken cancellationToken = default) + { + return memory.AddTodoAsync(new AddTodoRequest(project, task, priority, dueAt, details, tags, status), cancellationToken); + } + + [McpServerTool(Name = "AddNote", UseStructuredContent = true, Destructive = false)] + [Description("Add a general note or idea to an existing explicit project.")] + public static Task AddNote( + ICortexMemoryService memory, + [Description("Existing project name or slug. The project must already exist.")] string project, + [Description("Required note subject.")] string subject, + [Description("Required note body.")] string body, + [Description("Optional tags for filtering and recall.")] string[]? tags = null, + [Description("Optional item status. Defaults to active.")] string? status = null, + CancellationToken cancellationToken = default) + { + return memory.AddNoteAsync(new AddNoteRequest(project, subject, body, tags, status), cancellationToken); } [McpServerTool(Name = "FindItems", UseStructuredContent = true, ReadOnly = true, Destructive = false)] [Description("Find memory items with hybrid semantic and keyword search.")] public static Task> FindItems( ICortexMemoryService memory, - [Description("Natural language search query.")] string query, + [Description("One to four natural language search queries. Each query is embedded and searched separately.")] string[] queries, [Description("Project name or slug. Required unless searchAllProjects is true.")] string? project = null, [Description("Set true to search across all projects.")] bool searchAllProjects = false, [Description("Optional category filter: requirement, location, todo, note.")] string? category = null, [Description("Maximum number of results, from 1 to 50.")] int limit = 10, CancellationToken cancellationToken = default) { - return memory.FindItemsAsync(new FindMemoryItemsRequest(query, project, searchAllProjects, category, limit), cancellationToken); + return memory.FindItemsAsync(new FindMemoryItemsRequest(queries, project, searchAllProjects, category, limit), cancellationToken); } - [McpServerTool(Name = "UpdateItem", UseStructuredContent = true, Destructive = true)] - [Description("Update an existing memory item. Content-related changes automatically regenerate the embedding.")] - public static Task UpdateItem( + [McpServerTool(Name = "ListItemEmbeddings", UseStructuredContent = true, ReadOnly = true, Destructive = false)] + [Description("List default and additional embeddings for an existing memory item.")] + public static Task> ListItemEmbeddings( + ICortexMemoryService memory, + [Description("Existing project name or slug.")] string project, + [Description("Memory item id from find results.")] Guid itemId, + CancellationToken cancellationToken = default) + { + return memory.ListItemEmbeddingsAsync(new ListItemEmbeddingsRequest(project, itemId), cancellationToken); + } + + [McpServerTool(Name = "AddItemEmbedding", UseStructuredContent = true, Destructive = true)] + [Description("Add an additional discoverability embedding to an existing memory item. The canonical item content is unchanged.")] + public static Task AddItemEmbedding( + ICortexMemoryService memory, + [Description("Existing project name or slug.")] string project, + [Description("Memory item id from find results.")] Guid itemId, + [Description("Required text to embed for this specific discoverability aspect.")] string text, + [Description("Optional short label for what this embedding targets.")] string? label = null, + CancellationToken cancellationToken = default) + { + return memory.AddItemEmbeddingAsync(new AddItemEmbeddingRequest(project, itemId, label, text), cancellationToken); + } + + [McpServerTool(Name = "UpdateItemEmbedding", UseStructuredContent = true, Destructive = true)] + [Description("Update an additional item embedding. The default embedding cannot be edited directly.")] + public static Task UpdateItemEmbedding( + ICortexMemoryService memory, + [Description("Existing project name or slug.")] string project, + [Description("Memory item id from find results.")] Guid itemId, + [Description("Embedding id from ListItemEmbeddings.")] Guid embeddingId, + [Description("Replacement label. Pass an empty string to clear.")] string? label = null, + [Description("Replacement text to embed.")] string? text = null, + CancellationToken cancellationToken = default) + { + return memory.UpdateItemEmbeddingAsync(new UpdateItemEmbeddingRequest(project, itemId, embeddingId, label, text), cancellationToken); + } + + [McpServerTool(Name = "DeleteItemEmbedding", UseStructuredContent = true, Destructive = true)] + [Description("Delete an additional item embedding. The default embedding cannot be deleted.")] + public static Task DeleteItemEmbedding( + ICortexMemoryService memory, + [Description("Existing project name or slug.")] string project, + [Description("Memory item id from find results.")] Guid itemId, + [Description("Embedding id from ListItemEmbeddings.")] Guid embeddingId, + CancellationToken cancellationToken = default) + { + return memory.DeleteItemEmbeddingAsync(new DeleteItemEmbeddingRequest(project, itemId, embeddingId), cancellationToken); + } + + [McpServerTool(Name = "UpdateRequirement", UseStructuredContent = true, Destructive = true)] + [Description("Update an existing requirement item. Content-related changes automatically regenerate the embedding.")] + public static Task UpdateRequirement( ICortexMemoryService memory, [Description("Existing project name or slug.")] string project, [Description("Memory item id from find results.")] Guid id, - [Description("Optional new category: requirement, location, todo, note.")] string? category = null, - [Description("Optional new title.")] string? title = null, - [Description("Optional new content.")] string? content = null, + [Description("Replacement requirement, decision, or constraint statement.")] string? statement = null, + [Description("Replacement context explaining why this requirement matters.")] string? context = null, [Description("Optional replacement tag list.")] string[]? tags = null, - [Description("Optional replacement JSON metadata object.")] string? metadataJson = null, [Description("Optional new status.")] string? status = null, CancellationToken cancellationToken = default) { - return memory.UpdateItemAsync(new UpdateMemoryItemRequest(project, id, category, title, content, tags, metadataJson, status), cancellationToken); + return memory.UpdateRequirementAsync(new UpdateRequirementRequest(project, id, statement, context, tags, status), cancellationToken); + } + + [McpServerTool(Name = "UpdateLocation", UseStructuredContent = true, Destructive = true)] + [Description("Update an existing location item. Content-related changes automatically regenerate the embedding.")] + public static Task UpdateLocation( + ICortexMemoryService memory, + [Description("Existing project name or slug.")] string project, + [Description("Memory item id from find results.")] Guid id, + [Description("Replacement object, file, or resource name.")] string? itemName = null, + [Description("Replacement place where the item is located.")] string? place = null, + [Description("Replacement extra location details. Pass an empty string to clear.")] string? details = null, + [Description("Optional replacement tag list.")] string[]? tags = null, + [Description("Optional new status.")] string? status = null, + CancellationToken cancellationToken = default) + { + return memory.UpdateLocationAsync(new UpdateLocationRequest(project, id, itemName, place, details, tags, status), cancellationToken); + } + + [McpServerTool(Name = "UpdateTodo", UseStructuredContent = true, Destructive = true)] + [Description("Update an existing todo item. Content-related changes automatically regenerate the embedding.")] + public static Task UpdateTodo( + ICortexMemoryService memory, + [Description("Existing project name or slug.")] string project, + [Description("Memory item id from find results.")] Guid id, + [Description("Replacement action to complete.")] string? task = null, + [Description("Replacement priority.")] string? priority = null, + [Description("Replacement due timestamp.")] DateTimeOffset? dueAt = null, + [Description("Set true to clear the due timestamp.")] bool clearDueAt = false, + [Description("Replacement extra task details. Pass an empty string to clear.")] string? details = null, + [Description("Optional replacement tag list.")] string[]? tags = null, + [Description("Optional new status.")] string? status = null, + CancellationToken cancellationToken = default) + { + return memory.UpdateTodoAsync(new UpdateTodoRequest(project, id, task, priority, dueAt, clearDueAt, details, tags, status), cancellationToken); + } + + [McpServerTool(Name = "UpdateNote", UseStructuredContent = true, Destructive = true)] + [Description("Update an existing note item. Content-related changes automatically regenerate the embedding.")] + public static Task UpdateNote( + ICortexMemoryService memory, + [Description("Existing project name or slug.")] string project, + [Description("Memory item id from find results.")] Guid id, + [Description("Replacement note subject.")] string? subject = null, + [Description("Replacement note body.")] string? body = null, + [Description("Optional replacement tag list.")] string[]? tags = null, + [Description("Optional new status.")] string? status = null, + CancellationToken cancellationToken = default) + { + return memory.UpdateNoteAsync(new UpdateNoteRequest(project, id, subject, body, tags, status), cancellationToken); } [McpServerTool(Name = "DeleteItem", UseStructuredContent = true, Destructive = true)] diff --git a/src/Cortex.Core/Contracts/CortexDtos.cs b/src/Cortex.Core/Contracts/CortexDtos.cs index c715a60..2d24096 100644 --- a/src/Cortex.Core/Contracts/CortexDtos.cs +++ b/src/Cortex.Core/Contracts/CortexDtos.cs @@ -11,11 +11,12 @@ public sealed record MemoryItemDto( string Title, string Content, string[] Tags, - string MetadataJson, string Status, - string? EmbeddingModel, + string? DefaultEmbeddingModel, + int EmbeddingCount, DateTimeOffset CreatedAt, - DateTimeOffset UpdatedAt); + DateTimeOffset UpdatedAt, + MemoryItemDetailsDto Details); public sealed record SearchResultDto( Guid Id, @@ -24,7 +25,33 @@ public sealed record SearchResultDto( string Title, string Content, string[] Tags, - string MetadataJson, string Status, double Score, + int MatchedQueryIndex, + int MatchedEmbeddingPosition, + string? MatchedEmbeddingLabel, + DateTimeOffset UpdatedAt, + MemoryItemDetailsDto Details); + +public sealed record MemoryItemEmbeddingDto( + Guid Id, + Guid MemoryItemId, + int Position, + string? Label, + string Text, + string? EmbeddingModel, + DateTimeOffset CreatedAt, DateTimeOffset UpdatedAt); + +public sealed record MemoryItemDetailsDto( + string? RequirementStatement = null, + string? RequirementContext = null, + string? LocationItemName = null, + string? LocationPlace = null, + string? LocationDetails = null, + string? TodoTask = null, + string? TodoPriority = null, + DateTimeOffset? TodoDueAt = null, + string? TodoDetails = null, + string? NoteSubject = null, + string? NoteBody = null); diff --git a/src/Cortex.Core/Data/CortexDbContext.cs b/src/Cortex.Core/Data/CortexDbContext.cs index 277d623..911ae17 100644 --- a/src/Cortex.Core/Data/CortexDbContext.cs +++ b/src/Cortex.Core/Data/CortexDbContext.cs @@ -7,6 +7,11 @@ public sealed class CortexDbContext(DbContextOptions options) : { public DbSet Projects => Set(); public DbSet MemoryItems => Set(); + public DbSet MemoryItemEmbeddings => Set(); + public DbSet RequirementMemoryItems => Set(); + public DbSet LocationMemoryItems => Set(); + public DbSet TodoMemoryItems => Set(); + public DbSet NoteMemoryItems => Set(); protected override void OnModelCreating(ModelBuilder modelBuilder) { @@ -34,9 +39,7 @@ public sealed class CortexDbContext(DbContextOptions options) : entity.Property(item => item.Title).HasColumnName("title").HasMaxLength(240).IsRequired(); entity.Property(item => item.Content).HasColumnName("content").IsRequired(); entity.Property(item => item.Tags).HasColumnName("tags").HasColumnType("text[]"); - entity.Property(item => item.MetadataJson).HasColumnName("metadata").HasColumnType("jsonb"); entity.Property(item => item.Status).HasColumnName("status").HasMaxLength(40); - entity.Property(item => item.EmbeddingModel).HasColumnName("embedding_model").HasMaxLength(120); entity.Property(item => item.CreatedAt).HasColumnName("created_at"); entity.Property(item => item.UpdatedAt).HasColumnName("updated_at"); entity.Property(item => item.DeletedAt).HasColumnName("deleted_at"); @@ -44,5 +47,57 @@ public sealed class CortexDbContext(DbContextOptions options) : entity.HasIndex(item => new { item.ProjectId, item.Category }); entity.HasIndex(item => item.DeletedAt); }); + + modelBuilder.Entity(entity => + { + entity.ToTable("memory_item_embeddings", table => + { + table.HasCheckConstraint("ck_memory_item_embeddings_position_non_negative", "position >= 0"); + }); + entity.HasKey(embedding => embedding.Id); + entity.Property(embedding => embedding.Id).HasColumnName("id"); + entity.Property(embedding => embedding.MemoryItemId).HasColumnName("memory_item_id"); + entity.Property(embedding => embedding.Position).HasColumnName("position"); + entity.Property(embedding => embedding.Label).HasColumnName("label").HasMaxLength(120); + entity.Property(embedding => embedding.Text).HasColumnName("text").IsRequired(); + entity.Property(embedding => embedding.EmbeddingModel).HasColumnName("embedding_model").HasMaxLength(120); + entity.Property(embedding => embedding.CreatedAt).HasColumnName("created_at"); + entity.Property(embedding => embedding.UpdatedAt).HasColumnName("updated_at"); + entity.HasOne(embedding => embedding.MemoryItem) + .WithMany(item => item.Embeddings) + .HasForeignKey(embedding => embedding.MemoryItemId); + entity.HasIndex(embedding => new { embedding.MemoryItemId, embedding.Position }).IsUnique(); + }); + + modelBuilder.Entity(entity => + { + entity.ToTable("requirement_memory_items"); + entity.Property(item => item.Statement).HasColumnName("statement").IsRequired(); + entity.Property(item => item.Context).HasColumnName("context").IsRequired(); + }); + + modelBuilder.Entity(entity => + { + entity.ToTable("location_memory_items"); + entity.Property(item => item.ItemName).HasColumnName("item_name").HasMaxLength(240).IsRequired(); + entity.Property(item => item.Place).HasColumnName("place").HasMaxLength(400).IsRequired(); + entity.Property(item => item.Details).HasColumnName("details"); + }); + + modelBuilder.Entity(entity => + { + entity.ToTable("todo_memory_items"); + entity.Property(item => item.Task).HasColumnName("task").HasMaxLength(240).IsRequired(); + entity.Property(item => item.Priority).HasColumnName("priority").HasMaxLength(40).IsRequired(); + entity.Property(item => item.DueAt).HasColumnName("due_at"); + entity.Property(item => item.Details).HasColumnName("details"); + }); + + modelBuilder.Entity(entity => + { + entity.ToTable("note_memory_items"); + entity.Property(item => item.Subject).HasColumnName("subject").HasMaxLength(240).IsRequired(); + entity.Property(item => item.Body).HasColumnName("body").IsRequired(); + }); } } diff --git a/src/Cortex.Core/Data/CortexDbContextFactory.cs b/src/Cortex.Core/Data/CortexDbContextFactory.cs new file mode 100644 index 0000000..0085937 --- /dev/null +++ b/src/Cortex.Core/Data/CortexDbContextFactory.cs @@ -0,0 +1,23 @@ +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Design; + +namespace Cortex.Core.Data; + +public sealed class CortexDbContextFactory : IDesignTimeDbContextFactory +{ + private const string DefaultConnectionString = + "Host=localhost;Port=54329;Database=cortex;Username=cortex;Password=cortex_dev_password;GSS Encryption Mode=Disable"; + + public CortexDbContext CreateDbContext(string[] args) + { + var connectionString = Environment.GetEnvironmentVariable("ConnectionStrings__Cortex") + ?? Environment.GetEnvironmentVariable("CORTEX_CONNECTION_STRING") + ?? DefaultConnectionString; + + var options = new DbContextOptionsBuilder() + .UseNpgsql(connectionString) + .Options; + + return new CortexDbContext(options); + } +} diff --git a/src/Cortex.Core/Data/SeedData.cs b/src/Cortex.Core/Data/SeedData.cs index dc872ec..e44c65b 100644 --- a/src/Cortex.Core/Data/SeedData.cs +++ b/src/Cortex.Core/Data/SeedData.cs @@ -28,25 +28,21 @@ public static class SeedData return; } - await memory.AddItemAsync( - new AddMemoryItemRequest( + await memory.AddRequirementAsync( + new AddRequirementRequest( "cortex", - "requirement", "Cortex runs as an HTTP MCP server", "The Cortex MCP server is a .NET 10 ASP.NET Core app hosted by Kestrel and deployed locally with Docker.", ["mcp", "dotnet", "docker"], - """{"source":"seed"}""", "active"), cancellationToken); - await memory.AddItemAsync( - new AddMemoryItemRequest( + await memory.AddNoteAsync( + new AddNoteRequest( "cortex", - "note", "Embedding model choice", "Use BAAI/bge-small-en-v1.5 through a local Hugging Face Text Embeddings Inference container. The vector size is 384.", ["embeddings", "pgvector"], - """{"source":"seed"}""", "active"), cancellationToken); } diff --git a/src/Cortex.Core/Domain/MemoryItem.cs b/src/Cortex.Core/Domain/MemoryItem.cs index 2862580..90fe757 100644 --- a/src/Cortex.Core/Domain/MemoryItem.cs +++ b/src/Cortex.Core/Domain/MemoryItem.cs @@ -1,18 +1,77 @@ namespace Cortex.Core.Domain; -public sealed class MemoryItem : ITimestampedEntity +public abstract class MemoryItem : ITimestampedEntity { public Guid Id { get; set; } public Guid ProjectId { get; set; } public Project? Project { get; set; } - public MemoryCategory Category { get; set; } + public MemoryCategory Category { get; protected set; } public required string Title { get; set; } public required string Content { get; set; } public string[] Tags { get; set; } = []; - public string MetadataJson { get; set; } = "{}"; public string Status { get; set; } = "active"; - public string? EmbeddingModel { get; set; } + public ICollection Embeddings { get; set; } = []; public DateTimeOffset CreatedAt { get; set; } public DateTimeOffset UpdatedAt { get; set; } public DateTimeOffset? DeletedAt { get; set; } } + +public sealed class MemoryItemEmbedding : ITimestampedEntity +{ + public Guid Id { get; set; } + public Guid MemoryItemId { get; set; } + public MemoryItem? MemoryItem { get; set; } + public int Position { get; set; } + public string? Label { get; set; } + public required string Text { get; set; } + public string? EmbeddingModel { get; set; } + public DateTimeOffset CreatedAt { get; set; } + public DateTimeOffset UpdatedAt { get; set; } +} + +public sealed class RequirementMemoryItem : MemoryItem +{ + public RequirementMemoryItem() + { + Category = MemoryCategory.Requirement; + } + + public required string Statement { get; set; } + public required string Context { get; set; } +} + +public sealed class LocationMemoryItem : MemoryItem +{ + public LocationMemoryItem() + { + Category = MemoryCategory.Location; + } + + public required string ItemName { get; set; } + public required string Place { get; set; } + public string? Details { get; set; } +} + +public sealed class TodoMemoryItem : MemoryItem +{ + public TodoMemoryItem() + { + Category = MemoryCategory.Todo; + } + + public required string Task { get; set; } + public required string Priority { get; set; } + public DateTimeOffset? DueAt { get; set; } + public string? Details { get; set; } +} + +public sealed class NoteMemoryItem : MemoryItem +{ + public NoteMemoryItem() + { + Category = MemoryCategory.Note; + } + + public required string Subject { get; set; } + public required string Body { get; set; } +} diff --git a/src/Cortex.Core/Migrations/20260711153000_AddCategorySpecificMemoryItems.cs b/src/Cortex.Core/Migrations/20260711153000_AddCategorySpecificMemoryItems.cs new file mode 100644 index 0000000..6386fa5 --- /dev/null +++ b/src/Cortex.Core/Migrations/20260711153000_AddCategorySpecificMemoryItems.cs @@ -0,0 +1,149 @@ +using Cortex.Core.Data; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace Cortex.Core.Migrations; + +[DbContext(typeof(CortexDbContext))] +[Migration("20260711153000_AddCategorySpecificMemoryItems")] +public partial class AddCategorySpecificMemoryItems : Migration +{ + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateTable( + name: "requirement_memory_items", + columns: table => new + { + id = table.Column(type: "uuid", nullable: false), + statement = table.Column(type: "text", nullable: false), + context = table.Column(type: "text", nullable: false) + }, + constraints: table => + { + table.PrimaryKey("pk_requirement_memory_items", x => x.id); + table.ForeignKey( + name: "fk_requirement_memory_items_memory_items_id", + column: x => x.id, + principalTable: "memory_items", + principalColumn: "id", + onDelete: ReferentialAction.Cascade); + }); + + migrationBuilder.CreateTable( + name: "location_memory_items", + columns: table => new + { + id = table.Column(type: "uuid", nullable: false), + item_name = table.Column(type: "character varying(240)", maxLength: 240, nullable: false), + place = table.Column(type: "character varying(400)", maxLength: 400, nullable: false), + details = table.Column(type: "text", nullable: true) + }, + constraints: table => + { + table.PrimaryKey("pk_location_memory_items", x => x.id); + table.ForeignKey( + name: "fk_location_memory_items_memory_items_id", + column: x => x.id, + principalTable: "memory_items", + principalColumn: "id", + onDelete: ReferentialAction.Cascade); + }); + + migrationBuilder.CreateTable( + name: "todo_memory_items", + columns: table => new + { + id = table.Column(type: "uuid", nullable: false), + task = table.Column(type: "character varying(240)", maxLength: 240, nullable: false), + priority = table.Column(type: "character varying(40)", maxLength: 40, nullable: false), + due_at = table.Column(type: "timestamp with time zone", nullable: true), + details = table.Column(type: "text", nullable: true) + }, + constraints: table => + { + table.PrimaryKey("pk_todo_memory_items", x => x.id); + table.ForeignKey( + name: "fk_todo_memory_items_memory_items_id", + column: x => x.id, + principalTable: "memory_items", + principalColumn: "id", + onDelete: ReferentialAction.Cascade); + }); + + migrationBuilder.CreateTable( + name: "note_memory_items", + columns: table => new + { + id = table.Column(type: "uuid", nullable: false), + subject = table.Column(type: "character varying(240)", maxLength: 240, nullable: false), + body = table.Column(type: "text", nullable: false) + }, + constraints: table => + { + table.PrimaryKey("pk_note_memory_items", x => x.id); + table.ForeignKey( + name: "fk_note_memory_items_memory_items_id", + column: x => x.id, + principalTable: "memory_items", + principalColumn: "id", + onDelete: ReferentialAction.Cascade); + }); + + migrationBuilder.Sql(""" + INSERT INTO requirement_memory_items (id, statement, context) + SELECT id, title, content + FROM memory_items + WHERE category = 'Requirement'; + """); + + migrationBuilder.Sql(""" + INSERT INTO location_memory_items (id, item_name, place, details) + SELECT + id, + COALESCE(NULLIF(metadata->>'object', ''), title), + COALESCE(NULLIF(metadata->>'place', ''), content), + NULLIF(content, '') + FROM memory_items + WHERE category = 'Location'; + """); + + migrationBuilder.Sql(""" + INSERT INTO todo_memory_items (id, task, priority, details) + SELECT + id, + title, + COALESCE(NULLIF(metadata->>'priority', ''), 'normal'), + NULLIF(content, '') + FROM memory_items + WHERE category = 'Todo'; + """); + + migrationBuilder.Sql(""" + INSERT INTO note_memory_items (id, subject, body) + SELECT id, title, content + FROM memory_items + WHERE category = 'Note'; + """); + + migrationBuilder.DropColumn( + name: "metadata", + table: "memory_items"); + } + + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.AddColumn( + name: "metadata", + table: "memory_items", + type: "jsonb", + nullable: false, + defaultValue: "{}"); + + migrationBuilder.DropTable(name: "location_memory_items"); + migrationBuilder.DropTable(name: "note_memory_items"); + migrationBuilder.DropTable(name: "requirement_memory_items"); + migrationBuilder.DropTable(name: "todo_memory_items"); + } +} diff --git a/src/Cortex.Core/Migrations/20260711164500_AddMemoryItemEmbeddings.cs b/src/Cortex.Core/Migrations/20260711164500_AddMemoryItemEmbeddings.cs new file mode 100644 index 0000000..376d2fe --- /dev/null +++ b/src/Cortex.Core/Migrations/20260711164500_AddMemoryItemEmbeddings.cs @@ -0,0 +1,110 @@ +using Cortex.Core.Data; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace Cortex.Core.Migrations; + +[DbContext(typeof(CortexDbContext))] +[Migration("20260711164500_AddMemoryItemEmbeddings")] +public partial class AddMemoryItemEmbeddings : Migration +{ + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateTable( + name: "memory_item_embeddings", + columns: table => new + { + id = table.Column(type: "uuid", nullable: false), + memory_item_id = table.Column(type: "uuid", nullable: false), + position = table.Column(type: "integer", nullable: false), + label = table.Column(type: "character varying(120)", maxLength: 120, nullable: true), + text = table.Column(type: "text", nullable: false), + embedding_model = table.Column(type: "character varying(120)", maxLength: 120, nullable: true), + created_at = table.Column(type: "timestamp with time zone", nullable: false), + updated_at = table.Column(type: "timestamp with time zone", nullable: false) + }, + constraints: table => + { + table.PrimaryKey("pk_memory_item_embeddings", x => x.id); + table.ForeignKey( + name: "fk_memory_item_embeddings_memory_items_memory_item_id", + column: x => x.memory_item_id, + principalTable: "memory_items", + principalColumn: "id", + onDelete: ReferentialAction.Cascade); + }); + + migrationBuilder.Sql("ALTER TABLE memory_item_embeddings ADD COLUMN embedding vector(384);"); + migrationBuilder.Sql("ALTER TABLE memory_item_embeddings ADD CONSTRAINT ck_memory_item_embeddings_position_non_negative CHECK (position >= 0);"); + + migrationBuilder.CreateIndex( + name: "ix_memory_item_embeddings_memory_item_id_position", + table: "memory_item_embeddings", + columns: ["memory_item_id", "position"], + unique: true); + + migrationBuilder.Sql(""" + INSERT INTO memory_item_embeddings ( + id, + memory_item_id, + position, + label, + text, + embedding_model, + created_at, + updated_at) + SELECT + id, + id, + 0, + NULL, + category || E'\n' || title || E'\n' || content || E'\nTags: ' || array_to_string(tags, ', '), + embedding_model, + created_at, + updated_at + FROM memory_items; + """); + + migrationBuilder.Sql(""" + UPDATE memory_item_embeddings mie + SET embedding = mi.embedding + FROM memory_items mi + WHERE mie.memory_item_id = mi.id + AND mie.position = 0; + """); + + migrationBuilder.Sql("CREATE INDEX ix_memory_item_embeddings_embedding_hnsw ON memory_item_embeddings USING hnsw (embedding vector_cosine_ops);"); + migrationBuilder.Sql("DROP INDEX IF EXISTS ix_memory_items_embedding_hnsw;"); + migrationBuilder.Sql("ALTER TABLE memory_items DROP COLUMN embedding;"); + + migrationBuilder.DropColumn( + name: "embedding_model", + table: "memory_items"); + } + + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.AddColumn( + name: "embedding_model", + table: "memory_items", + type: "character varying(120)", + maxLength: 120, + nullable: true); + + migrationBuilder.Sql("ALTER TABLE memory_items ADD COLUMN embedding vector(384);"); + + migrationBuilder.Sql(""" + UPDATE memory_items mi + SET embedding_model = mie.embedding_model, + embedding = mie.embedding + FROM memory_item_embeddings mie + WHERE mie.memory_item_id = mi.id + AND mie.position = 0; + """); + + migrationBuilder.Sql("CREATE INDEX ix_memory_items_embedding_hnsw ON memory_items USING hnsw (embedding vector_cosine_ops);"); + migrationBuilder.DropTable(name: "memory_item_embeddings"); + } +} diff --git a/src/Cortex.Core/Migrations/CortexDbContextModelSnapshot.cs b/src/Cortex.Core/Migrations/CortexDbContextModelSnapshot.cs index 3437008..ae078a4 100644 --- a/src/Cortex.Core/Migrations/CortexDbContextModelSnapshot.cs +++ b/src/Cortex.Core/Migrations/CortexDbContextModelSnapshot.cs @@ -1,59 +1,327 @@ +// +using System; using Cortex.Core.Data; -using Cortex.Core.Domain; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Infrastructure; using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; #nullable disable -namespace Cortex.Core.Migrations; - -[DbContext(typeof(CortexDbContext))] -public sealed class CortexDbContextModelSnapshot : ModelSnapshot +namespace Cortex.Core.Migrations { - protected override void BuildModel(ModelBuilder modelBuilder) + [DbContext(typeof(CortexDbContext))] + partial class CortexDbContextModelSnapshot : ModelSnapshot { - modelBuilder - .HasAnnotation("ProductVersion", "10.0.0") - .HasPostgresExtension("vector"); - - modelBuilder.Entity("Cortex.Core.Domain.Project", entity => + protected override void BuildModel(ModelBuilder modelBuilder) { - entity.Property("Id").HasColumnName("id").HasColumnType("uuid"); - entity.Property("CreatedAt").HasColumnName("created_at").HasColumnType("timestamp with time zone"); - entity.Property("Name").IsRequired().HasMaxLength(160).HasColumnName("name").HasColumnType("character varying(160)"); - entity.Property("Slug").IsRequired().HasMaxLength(180).HasColumnName("slug").HasColumnType("character varying(180)"); - entity.Property("UpdatedAt").HasColumnName("updated_at").HasColumnType("timestamp with time zone"); - entity.HasKey("Id"); - entity.HasIndex("Slug").IsUnique(); - entity.ToTable("projects"); - }); +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.4") + .HasAnnotation("Relational:MaxIdentifierLength", 63); - modelBuilder.Entity("Cortex.Core.Domain.MemoryItem", entity => - { - entity.Property("Id").HasColumnName("id").HasColumnType("uuid"); - entity.Property("Category").HasColumnName("category").HasMaxLength(40).HasConversion(); - entity.Property("Content").IsRequired().HasColumnName("content").HasColumnType("text"); - entity.Property("CreatedAt").HasColumnName("created_at").HasColumnType("timestamp with time zone"); - entity.Property("DeletedAt").HasColumnName("deleted_at").HasColumnType("timestamp with time zone"); - entity.Property("EmbeddingModel").HasColumnName("embedding_model").HasMaxLength(120).HasColumnType("character varying(120)"); - entity.Property("MetadataJson").IsRequired().HasColumnName("metadata").HasColumnType("jsonb"); - entity.Property("ProjectId").HasColumnName("project_id").HasColumnType("uuid"); - entity.Property("Status").IsRequired().HasColumnName("status").HasMaxLength(40).HasColumnType("character varying(40)"); - entity.Property("Tags").IsRequired().HasColumnName("tags").HasColumnType("text[]"); - entity.Property("Title").IsRequired().HasColumnName("title").HasMaxLength(240).HasColumnType("character varying(240)"); - entity.Property("UpdatedAt").HasColumnName("updated_at").HasColumnType("timestamp with time zone"); - entity.HasKey("Id"); - entity.HasIndex("DeletedAt"); - entity.HasIndex("ProjectId", "Category"); - entity.HasOne("Cortex.Core.Domain.Project", "Project").WithMany("Items").HasForeignKey("ProjectId").OnDelete(DeleteBehavior.Cascade).IsRequired(); - entity.Navigation("Project"); - entity.ToTable("memory_items"); - }); + NpgsqlModelBuilderExtensions.HasPostgresExtension(modelBuilder, "vector"); + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); - modelBuilder.Entity("Cortex.Core.Domain.Project", entity => - { - entity.Navigation("Items"); - }); + modelBuilder.Entity("Cortex.Core.Domain.MemoryItem", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Category") + .IsRequired() + .HasMaxLength(40) + .HasColumnType("character varying(40)") + .HasColumnName("category"); + + b.Property("Content") + .IsRequired() + .HasColumnType("text") + .HasColumnName("content"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("DeletedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("deleted_at"); + + b.Property("ProjectId") + .HasColumnType("uuid") + .HasColumnName("project_id"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(40) + .HasColumnType("character varying(40)") + .HasColumnName("status"); + + b.PrimitiveCollection("Tags") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("tags"); + + b.Property("Title") + .IsRequired() + .HasMaxLength(240) + .HasColumnType("character varying(240)") + .HasColumnName("title"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.HasKey("Id"); + + b.HasIndex("DeletedAt"); + + b.HasIndex("ProjectId", "Category"); + + b.ToTable("memory_items", (string)null); + + b.UseTptMappingStrategy(); + }); + + modelBuilder.Entity("Cortex.Core.Domain.MemoryItemEmbedding", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("EmbeddingModel") + .HasMaxLength(120) + .HasColumnType("character varying(120)") + .HasColumnName("embedding_model"); + + b.Property("Label") + .HasMaxLength(120) + .HasColumnType("character varying(120)") + .HasColumnName("label"); + + b.Property("MemoryItemId") + .HasColumnType("uuid") + .HasColumnName("memory_item_id"); + + b.Property("Position") + .HasColumnType("integer") + .HasColumnName("position"); + + b.Property("Text") + .IsRequired() + .HasColumnType("text") + .HasColumnName("text"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.HasKey("Id"); + + b.HasIndex("MemoryItemId", "Position") + .IsUnique(); + + b.ToTable("memory_item_embeddings", null, t => + { + t.HasCheckConstraint("ck_memory_item_embeddings_position_non_negative", "position >= 0"); + }); + }); + + modelBuilder.Entity("Cortex.Core.Domain.Project", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(160) + .HasColumnType("character varying(160)") + .HasColumnName("name"); + + b.Property("Slug") + .IsRequired() + .HasMaxLength(180) + .HasColumnType("character varying(180)") + .HasColumnName("slug"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.HasKey("Id"); + + b.HasIndex("Slug") + .IsUnique(); + + b.ToTable("projects", (string)null); + }); + + modelBuilder.Entity("Cortex.Core.Domain.LocationMemoryItem", b => + { + b.HasBaseType("Cortex.Core.Domain.MemoryItem"); + + b.Property("Details") + .HasColumnType("text") + .HasColumnName("details"); + + b.Property("ItemName") + .IsRequired() + .HasMaxLength(240) + .HasColumnType("character varying(240)") + .HasColumnName("item_name"); + + b.Property("Place") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("character varying(400)") + .HasColumnName("place"); + + b.ToTable("location_memory_items", (string)null); + }); + + modelBuilder.Entity("Cortex.Core.Domain.NoteMemoryItem", b => + { + b.HasBaseType("Cortex.Core.Domain.MemoryItem"); + + b.Property("Body") + .IsRequired() + .HasColumnType("text") + .HasColumnName("body"); + + b.Property("Subject") + .IsRequired() + .HasMaxLength(240) + .HasColumnType("character varying(240)") + .HasColumnName("subject"); + + b.ToTable("note_memory_items", (string)null); + }); + + modelBuilder.Entity("Cortex.Core.Domain.RequirementMemoryItem", b => + { + b.HasBaseType("Cortex.Core.Domain.MemoryItem"); + + b.Property("Context") + .IsRequired() + .HasColumnType("text") + .HasColumnName("context"); + + b.Property("Statement") + .IsRequired() + .HasColumnType("text") + .HasColumnName("statement"); + + b.ToTable("requirement_memory_items", (string)null); + }); + + modelBuilder.Entity("Cortex.Core.Domain.TodoMemoryItem", b => + { + b.HasBaseType("Cortex.Core.Domain.MemoryItem"); + + b.Property("Details") + .HasColumnType("text") + .HasColumnName("details"); + + b.Property("DueAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("due_at"); + + b.Property("Priority") + .IsRequired() + .HasMaxLength(40) + .HasColumnType("character varying(40)") + .HasColumnName("priority"); + + b.Property("Task") + .IsRequired() + .HasMaxLength(240) + .HasColumnType("character varying(240)") + .HasColumnName("task"); + + b.ToTable("todo_memory_items", (string)null); + }); + + modelBuilder.Entity("Cortex.Core.Domain.MemoryItem", b => + { + b.HasOne("Cortex.Core.Domain.Project", "Project") + .WithMany("Items") + .HasForeignKey("ProjectId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + + b.Navigation("Project"); + }); + + modelBuilder.Entity("Cortex.Core.Domain.MemoryItemEmbedding", b => + { + b.HasOne("Cortex.Core.Domain.MemoryItem", "MemoryItem") + .WithMany("Embeddings") + .HasForeignKey("MemoryItemId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + + b.Navigation("MemoryItem"); + }); + + modelBuilder.Entity("Cortex.Core.Domain.LocationMemoryItem", b => + { + b.HasOne("Cortex.Core.Domain.MemoryItem", null) + .WithOne() + .HasForeignKey("Cortex.Core.Domain.LocationMemoryItem", "Id") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("Cortex.Core.Domain.NoteMemoryItem", b => + { + b.HasOne("Cortex.Core.Domain.MemoryItem", null) + .WithOne() + .HasForeignKey("Cortex.Core.Domain.NoteMemoryItem", "Id") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("Cortex.Core.Domain.RequirementMemoryItem", b => + { + b.HasOne("Cortex.Core.Domain.MemoryItem", null) + .WithOne() + .HasForeignKey("Cortex.Core.Domain.RequirementMemoryItem", "Id") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("Cortex.Core.Domain.TodoMemoryItem", b => + { + b.HasOne("Cortex.Core.Domain.MemoryItem", null) + .WithOne() + .HasForeignKey("Cortex.Core.Domain.TodoMemoryItem", "Id") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("Cortex.Core.Domain.MemoryItem", b => + { + b.Navigation("Embeddings"); + }); + + modelBuilder.Entity("Cortex.Core.Domain.Project", b => + { + b.Navigation("Items"); + }); +#pragma warning restore 612, 618 + } } } diff --git a/src/Cortex.Core/Services/CortexMemoryService.cs b/src/Cortex.Core/Services/CortexMemoryService.cs index e33ec31..aa44f74 100644 --- a/src/Cortex.Core/Services/CortexMemoryService.cs +++ b/src/Cortex.Core/Services/CortexMemoryService.cs @@ -1,5 +1,5 @@ using System.Data; -using System.Text.Json; +using System.Data.Common; using Cortex.Core.Contracts; using Cortex.Core.Data; using Cortex.Core.Domain; @@ -18,6 +18,8 @@ public sealed class CortexMemoryService( IOptions options, IConfiguration configuration) : ICortexMemoryService { + private const int MaxTitleLength = 240; + private const int MaxSearchQueries = 4; private readonly CortexOptions _options = options.Value; private readonly string _connectionString = configuration.GetConnectionString("Cortex") ?? throw new InvalidOperationException("Connection string 'Cortex' is missing."); @@ -59,32 +61,93 @@ public sealed class CortexMemoryService( .ToListAsync(cancellationToken); } - public async Task AddItemAsync(AddMemoryItemRequest request, CancellationToken cancellationToken) + public async Task AddRequirementAsync(AddRequirementRequest request, CancellationToken cancellationToken) { var project = await FindProjectAsync(request.Project, cancellationToken); - var category = ParseCategory(request.Category); - var metadataJson = NormalizeMetadata(request.MetadataJson); - var tags = NormalizeTags(request.Tags); + var statement = Required(request.Statement, nameof(request.Statement)); + var context = Required(request.Context, nameof(request.Context)); - var item = new MemoryItem + var item = new RequirementMemoryItem { Id = Guid.NewGuid(), ProjectId = project.Id, - Category = category, - Title = request.Title.Trim(), - Content = request.Content.Trim(), - Tags = tags, - MetadataJson = metadataJson, - Status = string.IsNullOrWhiteSpace(request.Status) ? "active" : request.Status.Trim(), - EmbeddingModel = _options.EmbeddingModel + Statement = statement, + Context = context, + Title = TitleFrom(statement), + Content = context, + Tags = NormalizeTags(request.Tags), + Status = NormalizeStatus(request.Status) }; - db.MemoryItems.Add(item); - await db.SaveChangesAsync(cancellationToken); - await UpdateEmbeddingAsync(item, cancellationToken); + return await AddAndEmbedAsync(item, project, cancellationToken); + } - item.Project = project; - return ToDto(item); + public async Task AddLocationAsync(AddLocationRequest request, CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + var itemName = Required(request.ItemName, nameof(request.ItemName)); + var place = Required(request.Place, nameof(request.Place)); + var details = Optional(request.Details); + + var item = new LocationMemoryItem + { + Id = Guid.NewGuid(), + ProjectId = project.Id, + ItemName = itemName, + Place = place, + Details = details, + Title = TitleFrom(itemName), + Content = BuildLocationContent(itemName, place, details), + Tags = NormalizeTags(request.Tags), + Status = NormalizeStatus(request.Status) + }; + + return await AddAndEmbedAsync(item, project, cancellationToken); + } + + public async Task AddTodoAsync(AddTodoRequest request, CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + var task = Required(request.Task, nameof(request.Task)); + var priority = Required(request.Priority, nameof(request.Priority)); + var details = Optional(request.Details); + + var item = new TodoMemoryItem + { + Id = Guid.NewGuid(), + ProjectId = project.Id, + Task = task, + Priority = priority, + DueAt = request.DueAt, + Details = details, + Title = TitleFrom(task), + Content = BuildTodoContent(task, priority, request.DueAt, details), + Tags = NormalizeTags(request.Tags), + Status = NormalizeStatus(request.Status) + }; + + return await AddAndEmbedAsync(item, project, cancellationToken); + } + + public async Task AddNoteAsync(AddNoteRequest request, CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + var subject = Required(request.Subject, nameof(request.Subject)); + var body = Required(request.Body, nameof(request.Body)); + + var item = new NoteMemoryItem + { + Id = Guid.NewGuid(), + ProjectId = project.Id, + Subject = subject, + Body = body, + Title = TitleFrom(subject), + Content = body, + Tags = NormalizeTags(request.Tags), + Status = NormalizeStatus(request.Status) + }; + + return await AddAndEmbedAsync(item, project, cancellationToken); } public async Task> FindItemsAsync(FindMemoryItemsRequest request, CancellationToken cancellationToken) @@ -94,44 +157,109 @@ public sealed class CortexMemoryService( throw new ArgumentException("Project is required unless search_all_projects is true.", nameof(request)); } + var queries = NormalizeQueries(request.Queries); var limit = Math.Clamp(request.Limit, 1, 50); - var queryEmbedding = VectorLiteral.From(await embeddings.EmbedAsync(request.Query, cancellationToken)); var projectSlug = string.IsNullOrWhiteSpace(request.Project) ? null : Slug.From(request.Project); var category = string.IsNullOrWhiteSpace(request.Category) ? null : ParseCategory(request.Category).ToString(); + var queryRows = string.Join(", ", queries.Select((_, index) => $"(@query{index}, @embedding{index}, {index})")); await using var connection = new NpgsqlConnection(_connectionString); await connection.OpenAsync(cancellationToken); await using var command = connection.CreateCommand(); - command.CommandText = """ + command.CommandText = $""" + WITH query_inputs(query_text, embedding_text, query_index) AS ( + VALUES {queryRows} + ), + scored AS ( + SELECT + mi.id, + p.slug AS project_slug, + mi.category, + mi.title, + mi.content, + mi.tags, + mi.status, + ( + COALESCE(ts_rank_cd(to_tsvector('simple', mi.title || ' ' || mi.content), plainto_tsquery('simple', qi.query_text)), 0) * 0.35 + + CASE + WHEN mie.embedding IS NULL THEN 0 + ELSE (1 - (mie.embedding <=> CAST(qi.embedding_text AS vector))) * 0.65 + END + )::double precision AS score, + qi.query_index, + mie.position AS embedding_position, + mie.label AS embedding_label, + mi.updated_at, + req.statement, + req.context, + loc.item_name, + loc.place, + loc.details, + todo.task, + todo.priority, + todo.due_at, + todo.details AS todo_details, + note.subject, + note.body + FROM memory_items mi + INNER JOIN projects p ON p.id = mi.project_id + INNER JOIN memory_item_embeddings mie ON mie.memory_item_id = mi.id + CROSS JOIN query_inputs qi + LEFT JOIN requirement_memory_items req ON req.id = mi.id + LEFT JOIN location_memory_items loc ON loc.id = mi.id + LEFT JOIN todo_memory_items todo ON todo.id = mi.id + LEFT JOIN note_memory_items note ON note.id = mi.id + WHERE mi.deleted_at IS NULL + AND (@search_all_projects OR p.slug = @project_slug) + AND (@category IS NULL OR mi.category = @category) + ), + ranked AS ( + SELECT + *, + ROW_NUMBER() OVER ( + PARTITION BY id + ORDER BY score DESC, embedding_position ASC, query_index ASC + ) AS match_rank + FROM scored + ) SELECT - mi.id, - p.slug AS project_slug, - mi.category, - mi.title, - mi.content, - mi.tags, - mi.metadata::text, - mi.status, - ( - COALESCE(ts_rank_cd(to_tsvector('simple', mi.title || ' ' || mi.content), plainto_tsquery('simple', @query)), 0) * 0.35 - + CASE - WHEN mi.embedding IS NULL THEN 0 - ELSE (1 - (mi.embedding <=> CAST(@embedding AS vector))) * 0.65 - END - )::double precision AS score, - mi.updated_at - FROM memory_items mi - INNER JOIN projects p ON p.id = mi.project_id - WHERE mi.deleted_at IS NULL - AND (@search_all_projects OR p.slug = @project_slug) - AND (@category IS NULL OR mi.category = @category) - ORDER BY score DESC, mi.updated_at DESC + id, + project_slug, + category, + title, + content, + tags, + status, + score, + query_index, + embedding_position, + embedding_label, + updated_at, + statement, + context, + item_name, + place, + details, + task, + priority, + due_at, + todo_details, + subject, + body + FROM ranked + WHERE match_rank = 1 + ORDER BY score DESC, updated_at DESC LIMIT @limit; """; - command.Parameters.Add("query", NpgsqlDbType.Text).Value = request.Query; - command.Parameters.Add("embedding", NpgsqlDbType.Text).Value = queryEmbedding; + for (var index = 0; index < queries.Length; index++) + { + command.Parameters.Add($"query{index}", NpgsqlDbType.Text).Value = queries[index]; + command.Parameters.Add($"embedding{index}", NpgsqlDbType.Text).Value = + VectorLiteral.From(await embeddings.EmbedAsync(queries[index], cancellationToken)); + } + command.Parameters.Add("search_all_projects", NpgsqlDbType.Boolean).Value = request.SearchAllProjects; command.Parameters.Add("project_slug", NpgsqlDbType.Text).Value = (object?)projectSlug ?? DBNull.Value; command.Parameters.Add("category", NpgsqlDbType.Text).Value = (object?)category ?? DBNull.Value; @@ -150,81 +278,285 @@ public sealed class CortexMemoryService( reader.GetString(4), reader.GetFieldValue(5), reader.GetString(6), - reader.GetString(7), - reader.GetDouble(8), - reader.GetFieldValue(9))); + reader.GetDouble(7), + reader.GetInt32(8), + reader.GetInt32(9), + NullableString(reader, 10), + reader.GetFieldValue(11), + new MemoryItemDetailsDto( + NullableString(reader, 12), + NullableString(reader, 13), + NullableString(reader, 14), + NullableString(reader, 15), + NullableString(reader, 16), + NullableString(reader, 17), + NullableString(reader, 18), + NullableDateTimeOffset(reader, 19), + NullableString(reader, 20), + NullableString(reader, 21), + NullableString(reader, 22)))); } return results; } - public async Task UpdateItemAsync(UpdateMemoryItemRequest request, CancellationToken cancellationToken) + public async Task> ListItemEmbeddingsAsync( + ListItemEmbeddingsRequest request, + CancellationToken cancellationToken) { var project = await FindProjectAsync(request.Project, cancellationToken); - var item = await FindActiveItemAsync(project.Id, request.Id, cancellationToken); + _ = await FindActiveItemAsync(project.Id, request.ItemId, cancellationToken); + + return await db.MemoryItemEmbeddings + .Where(embedding => embedding.MemoryItemId == request.ItemId) + .OrderBy(embedding => embedding.Position) + .Select(embedding => ToDto(embedding)) + .ToListAsync(cancellationToken); + } + + public async Task AddItemEmbeddingAsync( + AddItemEmbeddingRequest request, + CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + _ = await FindActiveItemAsync(project.Id, request.ItemId, cancellationToken); + var text = Required(request.Text, nameof(request.Text)); + + var maxPosition = await db.MemoryItemEmbeddings + .Where(embedding => embedding.MemoryItemId == request.ItemId) + .Select(embedding => (int?)embedding.Position) + .MaxAsync(cancellationToken) ?? 0; + + var embedding = new MemoryItemEmbedding + { + Id = Guid.NewGuid(), + MemoryItemId = request.ItemId, + Position = Math.Max(1, maxPosition + 1), + Label = Optional(request.Label), + Text = text, + EmbeddingModel = _options.EmbeddingModel + }; + + db.MemoryItemEmbeddings.Add(embedding); + await db.SaveChangesAsync(cancellationToken); + await UpdateEmbeddingVectorAsync(embedding, text, cancellationToken); + + return ToDto(embedding); + } + + public async Task UpdateItemEmbeddingAsync( + UpdateItemEmbeddingRequest request, + CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + _ = await FindActiveItemAsync(project.Id, request.ItemId, cancellationToken); + var embedding = await FindItemEmbeddingAsync(request.ItemId, request.EmbeddingId, cancellationToken); + + if (embedding.Position == 0) + { + throw new InvalidOperationException("The default embedding is generated from the item and cannot be edited directly."); + } + var shouldRegenerateEmbedding = false; - if (!string.IsNullOrWhiteSpace(request.Category)) + if (request.Label is not null) { - item.Category = ParseCategory(request.Category); + embedding.Label = Optional(request.Label); + } + + if (request.Text is not null) + { + embedding.Text = Required(request.Text, nameof(request.Text)); + embedding.EmbeddingModel = _options.EmbeddingModel; shouldRegenerateEmbedding = true; } - if (!string.IsNullOrWhiteSpace(request.Title)) - { - item.Title = request.Title.Trim(); - shouldRegenerateEmbedding = true; - } - - if (!string.IsNullOrWhiteSpace(request.Content)) - { - item.Content = request.Content.Trim(); - shouldRegenerateEmbedding = true; - } - - if (request.Tags is not null) - { - item.Tags = NormalizeTags(request.Tags); - shouldRegenerateEmbedding = true; - } - - if (request.MetadataJson is not null) - { - item.MetadataJson = NormalizeMetadata(request.MetadataJson); - } - - if (!string.IsNullOrWhiteSpace(request.Status)) - { - item.Status = request.Status.Trim(); - } - - if (shouldRegenerateEmbedding) - { - item.EmbeddingModel = _options.EmbeddingModel; - } - await db.SaveChangesAsync(cancellationToken); if (shouldRegenerateEmbedding) { - await UpdateEmbeddingAsync(item, cancellationToken); + await UpdateEmbeddingVectorAsync(embedding, embedding.Text, cancellationToken); } - item.Project = project; - return ToDto(item); + return ToDto(embedding); + } + + public async Task DeleteItemEmbeddingAsync( + DeleteItemEmbeddingRequest request, + CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + _ = await FindActiveItemAsync(project.Id, request.ItemId, cancellationToken); + var embedding = await FindItemEmbeddingAsync(request.ItemId, request.EmbeddingId, cancellationToken); + + if (embedding.Position == 0) + { + throw new InvalidOperationException("The default embedding cannot be deleted."); + } + + db.MemoryItemEmbeddings.Remove(embedding); + await db.SaveChangesAsync(cancellationToken); + return ToDto(embedding); + } + + public async Task UpdateRequirementAsync(UpdateRequirementRequest request, CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + var item = await FindActiveItemAsync(project.Id, request.Id, cancellationToken); + var shouldRegenerateEmbedding = false; + + if (!string.IsNullOrWhiteSpace(request.Statement)) + { + item.Statement = request.Statement.Trim(); + shouldRegenerateEmbedding = true; + } + + if (!string.IsNullOrWhiteSpace(request.Context)) + { + item.Context = request.Context.Trim(); + shouldRegenerateEmbedding = true; + } + + item.Title = TitleFrom(item.Statement); + item.Content = item.Context; + shouldRegenerateEmbedding |= ApplyCommonUpdates(item, request.Tags, request.Status); + + return await SaveUpdatedAsync(item, project, shouldRegenerateEmbedding, cancellationToken); + } + + public async Task UpdateLocationAsync(UpdateLocationRequest request, CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + var item = await FindActiveItemAsync(project.Id, request.Id, cancellationToken); + var shouldRegenerateEmbedding = false; + + if (!string.IsNullOrWhiteSpace(request.ItemName)) + { + item.ItemName = request.ItemName.Trim(); + shouldRegenerateEmbedding = true; + } + + if (!string.IsNullOrWhiteSpace(request.Place)) + { + item.Place = request.Place.Trim(); + shouldRegenerateEmbedding = true; + } + + if (request.Details is not null) + { + item.Details = Optional(request.Details); + shouldRegenerateEmbedding = true; + } + + item.Title = TitleFrom(item.ItemName); + item.Content = BuildLocationContent(item.ItemName, item.Place, item.Details); + shouldRegenerateEmbedding |= ApplyCommonUpdates(item, request.Tags, request.Status); + + return await SaveUpdatedAsync(item, project, shouldRegenerateEmbedding, cancellationToken); + } + + public async Task UpdateTodoAsync(UpdateTodoRequest request, CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + var item = await FindActiveItemAsync(project.Id, request.Id, cancellationToken); + var shouldRegenerateEmbedding = false; + + if (!string.IsNullOrWhiteSpace(request.Task)) + { + item.Task = request.Task.Trim(); + shouldRegenerateEmbedding = true; + } + + if (!string.IsNullOrWhiteSpace(request.Priority)) + { + item.Priority = request.Priority.Trim(); + shouldRegenerateEmbedding = true; + } + + if (request.ClearDueAt) + { + item.DueAt = null; + shouldRegenerateEmbedding = true; + } + else if (request.DueAt is not null) + { + item.DueAt = request.DueAt; + shouldRegenerateEmbedding = true; + } + + if (request.Details is not null) + { + item.Details = Optional(request.Details); + shouldRegenerateEmbedding = true; + } + + item.Title = TitleFrom(item.Task); + item.Content = BuildTodoContent(item.Task, item.Priority, item.DueAt, item.Details); + shouldRegenerateEmbedding |= ApplyCommonUpdates(item, request.Tags, request.Status); + + return await SaveUpdatedAsync(item, project, shouldRegenerateEmbedding, cancellationToken); + } + + public async Task UpdateNoteAsync(UpdateNoteRequest request, CancellationToken cancellationToken) + { + var project = await FindProjectAsync(request.Project, cancellationToken); + var item = await FindActiveItemAsync(project.Id, request.Id, cancellationToken); + var shouldRegenerateEmbedding = false; + + if (!string.IsNullOrWhiteSpace(request.Subject)) + { + item.Subject = request.Subject.Trim(); + shouldRegenerateEmbedding = true; + } + + if (!string.IsNullOrWhiteSpace(request.Body)) + { + item.Body = request.Body.Trim(); + shouldRegenerateEmbedding = true; + } + + item.Title = TitleFrom(item.Subject); + item.Content = item.Body; + shouldRegenerateEmbedding |= ApplyCommonUpdates(item, request.Tags, request.Status); + + return await SaveUpdatedAsync(item, project, shouldRegenerateEmbedding, cancellationToken); } public async Task DeleteItemAsync(DeleteMemoryItemRequest request, CancellationToken cancellationToken) { var project = await FindProjectAsync(request.Project, cancellationToken); - var item = await FindActiveItemAsync(project.Id, request.Id, cancellationToken); + var item = await FindActiveItemAsync(project.Id, request.Id, cancellationToken); item.DeletedAt = DateTimeOffset.UtcNow; item.Status = "deleted"; await db.SaveChangesAsync(cancellationToken); item.Project = project; - return ToDto(item); + return await ToDtoAsync(item, cancellationToken); + } + + private async Task AddAndEmbedAsync(T item, Project project, CancellationToken cancellationToken) + where T : MemoryItem + { + db.Set().Add(item); + await db.SaveChangesAsync(cancellationToken); + await UpsertDefaultEmbeddingAsync(item, cancellationToken); + + item.Project = project; + return await ToDtoAsync(item, cancellationToken); + } + + private async Task SaveUpdatedAsync(MemoryItem item, Project project, bool shouldRegenerateEmbedding, CancellationToken cancellationToken) + { + await db.SaveChangesAsync(cancellationToken); + + if (shouldRegenerateEmbedding) + { + await UpsertDefaultEmbeddingAsync(item, cancellationToken); + } + + item.Project = project; + return await ToDtoAsync(item, cancellationToken); } private async Task FindProjectAsync(string projectNameOrSlug, CancellationToken cancellationToken) @@ -240,18 +572,58 @@ public sealed class CortexMemoryService( return project; } - private async Task FindActiveItemAsync(Guid projectId, Guid itemId, CancellationToken cancellationToken) + private async Task FindActiveItemAsync(Guid projectId, Guid itemId, CancellationToken cancellationToken) + where T : MemoryItem { - var item = await db.MemoryItems.SingleOrDefaultAsync( + var item = await db.Set().SingleOrDefaultAsync( memory => memory.ProjectId == projectId && memory.Id == itemId && memory.DeletedAt == null, cancellationToken); - return item ?? throw new InvalidOperationException($"Memory item '{itemId}' does not exist in that project."); + return item ?? throw new InvalidOperationException($"Memory item '{itemId}' does not exist in that project or category."); } - private async Task UpdateEmbeddingAsync(MemoryItem item, CancellationToken cancellationToken) + private async Task FindItemEmbeddingAsync(Guid itemId, Guid embeddingId, CancellationToken cancellationToken) + { + var embedding = await db.MemoryItemEmbeddings.SingleOrDefaultAsync( + candidate => candidate.MemoryItemId == itemId && candidate.Id == embeddingId, + cancellationToken); + + return embedding ?? throw new InvalidOperationException($"Embedding '{embeddingId}' does not exist for that item."); + } + + private async Task UpsertDefaultEmbeddingAsync(MemoryItem item, CancellationToken cancellationToken) + { + var text = GetEmbeddingText(item); + var embedding = await db.MemoryItemEmbeddings.SingleOrDefaultAsync( + candidate => candidate.MemoryItemId == item.Id && candidate.Position == 0, + cancellationToken); + + if (embedding is null) + { + embedding = new MemoryItemEmbedding + { + Id = item.Id, + MemoryItemId = item.Id, + Position = 0, + Text = text, + EmbeddingModel = _options.EmbeddingModel + }; + + db.MemoryItemEmbeddings.Add(embedding); + } + else + { + embedding.Text = text; + embedding.EmbeddingModel = _options.EmbeddingModel; + } + + await db.SaveChangesAsync(cancellationToken); + await UpdateEmbeddingVectorAsync(embedding, text, cancellationToken); + return embedding; + } + + private async Task UpdateEmbeddingVectorAsync(MemoryItemEmbedding embedding, string text, CancellationToken cancellationToken) { - var text = $"{item.Category}\n{item.Title}\n{item.Content}\nTags: {string.Join(", ", item.Tags)}"; var vector = VectorLiteral.From(await embeddings.EmbedAsync(text, cancellationToken)); var updatedAt = DateTimeOffset.UtcNow; @@ -260,7 +632,7 @@ public sealed class CortexMemoryService( await using var command = connection.CreateCommand(); command.CommandText = """ - UPDATE memory_items + UPDATE memory_item_embeddings SET embedding = CAST(@embedding AS vector), embedding_model = @embedding_model, updated_at = @updated_at @@ -270,10 +642,11 @@ public sealed class CortexMemoryService( command.Parameters.AddWithValue("embedding", vector); command.Parameters.AddWithValue("embedding_model", _options.EmbeddingModel); command.Parameters.AddWithValue("updated_at", updatedAt); - command.Parameters.AddWithValue("id", item.Id); + command.Parameters.AddWithValue("id", embedding.Id); await command.ExecuteNonQueryAsync(cancellationToken); - item.UpdatedAt = updatedAt; + embedding.EmbeddingModel = _options.EmbeddingModel; + embedding.UpdatedAt = updatedAt; } private static MemoryCategory ParseCategory(string category) @@ -291,6 +664,44 @@ public sealed class CortexMemoryService( throw new ArgumentException("Category must be one of: requirement, location, todo, note.", nameof(category)); } + private static string[] NormalizeQueries(string[]? queries) + { + var normalized = queries? + .Select(query => query.Trim()) + .Where(query => query.Length > 0) + .ToArray() ?? []; + + if (normalized.Length == 0) + { + throw new ArgumentException("At least one query is required.", nameof(queries)); + } + + if (normalized.Length > MaxSearchQueries) + { + throw new ArgumentException($"FindItems supports at most {MaxSearchQueries} queries.", nameof(queries)); + } + + return normalized; + } + + private static bool ApplyCommonUpdates(MemoryItem item, string[]? tags, string? status) + { + var shouldRegenerateEmbedding = false; + + if (tags is not null) + { + item.Tags = NormalizeTags(tags); + shouldRegenerateEmbedding = true; + } + + if (!string.IsNullOrWhiteSpace(status)) + { + item.Status = status.Trim(); + } + + return shouldRegenerateEmbedding; + } + private static string[] NormalizeTags(string[]? tags) { return tags? @@ -300,15 +711,67 @@ public sealed class CortexMemoryService( .ToArray() ?? []; } - private static string NormalizeMetadata(string? metadataJson) + private static string NormalizeStatus(string? status) { - if (string.IsNullOrWhiteSpace(metadataJson)) + return string.IsNullOrWhiteSpace(status) ? "active" : status.Trim(); + } + + private static string Required(string value, string name) + { + if (string.IsNullOrWhiteSpace(value)) { - return "{}"; + throw new ArgumentException($"{name} is required.", name); } - using var document = JsonDocument.Parse(metadataJson); - return document.RootElement.GetRawText(); + return value.Trim(); + } + + private static string? Optional(string? value) + { + return string.IsNullOrWhiteSpace(value) ? null : value.Trim(); + } + + private static string TitleFrom(string value) + { + var trimmed = value.Trim(); + return trimmed.Length <= MaxTitleLength ? trimmed : $"{trimmed[..(MaxTitleLength - 3)]}..."; + } + + private static string BuildLocationContent(string itemName, string place, string? details) + { + var content = $"{itemName} is in {place}."; + return string.IsNullOrWhiteSpace(details) ? content : $"{content}\n{details}"; + } + + private static string BuildTodoContent(string task, string priority, DateTimeOffset? dueAt, string? details) + { + var due = dueAt is null ? null : $"\nDue: {dueAt.Value:O}"; + var body = $"Task: {task}\nPriority: {priority}{due}"; + return string.IsNullOrWhiteSpace(details) ? body : $"{body}\n{details}"; + } + + private static string GetEmbeddingText(MemoryItem item) + { + var details = item switch + { + RequirementMemoryItem requirement => $"Statement: {requirement.Statement}\nContext: {requirement.Context}", + LocationMemoryItem location => $"Item: {location.ItemName}\nPlace: {location.Place}\nDetails: {location.Details}", + TodoMemoryItem todo => $"Task: {todo.Task}\nPriority: {todo.Priority}\nDue: {todo.DueAt:O}\nDetails: {todo.Details}", + NoteMemoryItem note => $"Subject: {note.Subject}\nBody: {note.Body}", + _ => item.Content + }; + + return $"{item.Category}\n{item.Title}\n{item.Content}\n{details}\nTags: {string.Join(", ", item.Tags)}"; + } + + private static string? NullableString(DbDataReader reader, int ordinal) + { + return reader.IsDBNull(ordinal) ? null : reader.GetString(ordinal); + } + + private static DateTimeOffset? NullableDateTimeOffset(DbDataReader reader, int ordinal) + { + return reader.IsDBNull(ordinal) ? null : reader.GetFieldValue(ordinal); } private static ProjectDto ToDto(Project project) @@ -316,8 +779,13 @@ public sealed class CortexMemoryService( return new ProjectDto(project.Id, project.Name, project.Slug, project.CreatedAt); } - private static MemoryItemDto ToDto(MemoryItem item) + private async Task ToDtoAsync(MemoryItem item, CancellationToken cancellationToken) { + var embeddings = await db.MemoryItemEmbeddings + .Where(embedding => embedding.MemoryItemId == item.Id) + .Select(embedding => new { embedding.Position, embedding.EmbeddingModel }) + .ToListAsync(cancellationToken); + return new MemoryItemDto( item.Id, item.Project?.Slug ?? string.Empty, @@ -325,10 +793,47 @@ public sealed class CortexMemoryService( item.Title, item.Content, item.Tags, - item.MetadataJson, item.Status, - item.EmbeddingModel, + embeddings.SingleOrDefault(embedding => embedding.Position == 0)?.EmbeddingModel, + embeddings.Count, item.CreatedAt, - item.UpdatedAt); + item.UpdatedAt, + DetailsFrom(item)); + } + + private static MemoryItemEmbeddingDto ToDto(MemoryItemEmbedding embedding) + { + return new MemoryItemEmbeddingDto( + embedding.Id, + embedding.MemoryItemId, + embedding.Position, + embedding.Label, + embedding.Text, + embedding.EmbeddingModel, + embedding.CreatedAt, + embedding.UpdatedAt); + } + + private static MemoryItemDetailsDto DetailsFrom(MemoryItem item) + { + return item switch + { + RequirementMemoryItem requirement => new MemoryItemDetailsDto( + RequirementStatement: requirement.Statement, + RequirementContext: requirement.Context), + LocationMemoryItem location => new MemoryItemDetailsDto( + LocationItemName: location.ItemName, + LocationPlace: location.Place, + LocationDetails: location.Details), + TodoMemoryItem todo => new MemoryItemDetailsDto( + TodoTask: todo.Task, + TodoPriority: todo.Priority, + TodoDueAt: todo.DueAt, + TodoDetails: todo.Details), + NoteMemoryItem note => new MemoryItemDetailsDto( + NoteSubject: note.Subject, + NoteBody: note.Body), + _ => new MemoryItemDetailsDto() + }; } } diff --git a/src/Cortex.Core/Services/ICortexMemoryService.cs b/src/Cortex.Core/Services/ICortexMemoryService.cs index e289028..dd239a9 100644 --- a/src/Cortex.Core/Services/ICortexMemoryService.cs +++ b/src/Cortex.Core/Services/ICortexMemoryService.cs @@ -6,36 +6,120 @@ public interface ICortexMemoryService { Task CreateProjectAsync(string name, CancellationToken cancellationToken); Task> ListProjectsAsync(CancellationToken cancellationToken); - Task AddItemAsync(AddMemoryItemRequest request, CancellationToken cancellationToken); + Task AddRequirementAsync(AddRequirementRequest request, CancellationToken cancellationToken); + Task AddLocationAsync(AddLocationRequest request, CancellationToken cancellationToken); + Task AddTodoAsync(AddTodoRequest request, CancellationToken cancellationToken); + Task AddNoteAsync(AddNoteRequest request, CancellationToken cancellationToken); Task> FindItemsAsync(FindMemoryItemsRequest request, CancellationToken cancellationToken); - Task UpdateItemAsync(UpdateMemoryItemRequest request, CancellationToken cancellationToken); + Task> ListItemEmbeddingsAsync(ListItemEmbeddingsRequest request, CancellationToken cancellationToken); + Task AddItemEmbeddingAsync(AddItemEmbeddingRequest request, CancellationToken cancellationToken); + Task UpdateItemEmbeddingAsync(UpdateItemEmbeddingRequest request, CancellationToken cancellationToken); + Task DeleteItemEmbeddingAsync(DeleteItemEmbeddingRequest request, CancellationToken cancellationToken); + Task UpdateRequirementAsync(UpdateRequirementRequest request, CancellationToken cancellationToken); + Task UpdateLocationAsync(UpdateLocationRequest request, CancellationToken cancellationToken); + Task UpdateTodoAsync(UpdateTodoRequest request, CancellationToken cancellationToken); + Task UpdateNoteAsync(UpdateNoteRequest request, CancellationToken cancellationToken); Task DeleteItemAsync(DeleteMemoryItemRequest request, CancellationToken cancellationToken); } -public sealed record AddMemoryItemRequest( +public sealed record AddRequirementRequest( string Project, - string Category, - string Title, - string Content, + string Statement, + string Context, + string[]? Tags, + string? Status); + +public sealed record AddLocationRequest( + string Project, + string ItemName, + string Place, + string? Details, + string[]? Tags, + string? Status); + +public sealed record AddTodoRequest( + string Project, + string Task, + string Priority, + DateTimeOffset? DueAt, + string? Details, + string[]? Tags, + string? Status); + +public sealed record AddNoteRequest( + string Project, + string Subject, + string Body, string[]? Tags, - string? MetadataJson, string? Status); public sealed record FindMemoryItemsRequest( - string Query, + string[] Queries, string? Project, bool SearchAllProjects, string? Category, - int Limit); + int Limit) +{ + public FindMemoryItemsRequest(string query, string? project, bool searchAllProjects, string? category, int limit) + : this([query], project, searchAllProjects, category, limit) + { + } +} -public sealed record UpdateMemoryItemRequest( +public sealed record ListItemEmbeddingsRequest(string Project, Guid ItemId); + +public sealed record AddItemEmbeddingRequest( + string Project, + Guid ItemId, + string? Label, + string Text); + +public sealed record UpdateItemEmbeddingRequest( + string Project, + Guid ItemId, + Guid EmbeddingId, + string? Label, + string? Text); + +public sealed record DeleteItemEmbeddingRequest( + string Project, + Guid ItemId, + Guid EmbeddingId); + +public sealed record UpdateRequirementRequest( string Project, Guid Id, - string? Category, - string? Title, - string? Content, + string? Statement, + string? Context, + string[]? Tags, + string? Status); + +public sealed record UpdateLocationRequest( + string Project, + Guid Id, + string? ItemName, + string? Place, + string? Details, + string[]? Tags, + string? Status); + +public sealed record UpdateTodoRequest( + string Project, + Guid Id, + string? Task, + string? Priority, + DateTimeOffset? DueAt, + bool ClearDueAt, + string? Details, + string[]? Tags, + string? Status); + +public sealed record UpdateNoteRequest( + string Project, + Guid Id, + string? Subject, + string? Body, string[]? Tags, - string? MetadataJson, string? Status); public sealed record DeleteMemoryItemRequest(string Project, Guid Id); diff --git a/src/Cortex.Web/Components/Pages/Home.razor b/src/Cortex.Web/Components/Pages/Home.razor index 8f7d515..d9dbb7d 100644 --- a/src/Cortex.Web/Components/Pages/Home.razor +++ b/src/Cortex.Web/Components/Pages/Home.razor @@ -61,46 +61,92 @@ }
-
+

Add memory

@(SelectedProjectRequired ? "Choose a project first" : selectedProjectSlug)
-
- - - -
- -
- + @switch (addCategory) + { + case "requirement": + + + break; + case "location": +
+ + +
+ + break; + case "todo": +
+ + +
+ + + break; + default: + + + break; + } - -
+
@@ -119,8 +165,8 @@
@@ -163,10 +209,12 @@
@result.ProjectSlug @result.Category + q@(result.MatchedQueryIndex + 1) + e@result.MatchedEmbeddingPosition @result.Score.ToString("0.000")

@result.Title

-

@result.Content

+

@DetailSummary(result.Category, result.Details)

@foreach (var tag in result.Tags) { @@ -174,7 +222,7 @@ }
- +
@@ -219,10 +267,11 @@
@item.ProjectSlug @item.Category + @item.EmbeddingCount emb @item.UpdatedAt.LocalDateTime.ToString("g")

@item.Title

-

@item.Content

+

@DetailSummary(item.Category, item.Details)

@foreach (var tag in item.Tags) { @@ -230,7 +279,7 @@ }
- +
@@ -251,38 +300,112 @@
-
- - - -
- -
- - -
+ @switch (editCategory) + { + case "requirement": + + + break; + case "location": +
+ + +
+ + break; + case "todo": +
+ + +
+ + + break; + default: + + + break; + } + + + +
+
+

Embeddings

+ @editEmbeddings.Count total +
+ + @foreach (var embedding in editEmbeddings) + { +
+ + @embedding.Position: @(string.IsNullOrWhiteSpace(embedding.Label) ? "default" : embedding.Label) + + @if (embedding.Position > 0) + { + + } +
+ } + +
+ + +
+ +
@@ -295,6 +418,7 @@ @code { private static readonly string[] Categories = ["requirement", "location", "todo", "note"]; + private static readonly string[] Priorities = ["normal", "low", "high", "urgent"]; private IReadOnlyList projects = []; private IReadOnlyList recentItems = []; @@ -303,11 +427,19 @@ private string? selectedProjectSlug; private string newProjectName = string.Empty; private string addCategory = "note"; - private string addTitle = string.Empty; - private string addContent = string.Empty; + private string addRequirementStatement = string.Empty; + private string addRequirementContext = string.Empty; + private string addLocationItemName = string.Empty; + private string addLocationPlace = string.Empty; + private string addLocationDetails = string.Empty; + private string addTodoTask = string.Empty; + private string addTodoPriority = "normal"; + private string addTodoDueAt = string.Empty; + private string addTodoDetails = string.Empty; + private string addNoteSubject = string.Empty; + private string addNoteBody = string.Empty; private string addTags = string.Empty; - private string addMetadata = """{"source":"web"}"""; - private string searchQuery = string.Empty; + private string searchQueries = string.Empty; private string searchCategory = string.Empty; private bool searchAllProjects; private int searchLimit = 10; @@ -317,10 +449,21 @@ private bool isError; private EditState? editing; private string editCategory = "note"; - private string editTitle = string.Empty; - private string editContent = string.Empty; + private string editRequirementStatement = string.Empty; + private string editRequirementContext = string.Empty; + private string editLocationItemName = string.Empty; + private string editLocationPlace = string.Empty; + private string editLocationDetails = string.Empty; + private string editTodoTask = string.Empty; + private string editTodoPriority = "normal"; + private string editTodoDueAt = string.Empty; + private string editTodoDetails = string.Empty; + private string editNoteSubject = string.Empty; + private string editNoteBody = string.Empty; private string editTags = string.Empty; - private string editMetadata = "{}"; + private IReadOnlyList editEmbeddings = []; + private string newEmbeddingLabel = string.Empty; + private string newEmbeddingText = string.Empty; private bool SelectedProjectRequired => string.IsNullOrWhiteSpace(selectedProjectSlug); private string CurrentScopeTitle => string.IsNullOrWhiteSpace(selectedProjectSlug) ? "All projects" : selectedProjectSlug; @@ -369,7 +512,7 @@ }); } - private async Task AddItemAsync() + private async Task AddMemoryAsync() { if (string.IsNullOrWhiteSpace(selectedProjectSlug)) { @@ -379,16 +522,21 @@ await GuardedAsync(async () => { - var item = await Memory.AddItemAsync( - new AddMemoryItemRequest( - selectedProjectSlug, - addCategory, - addTitle, - addContent, - ParseTags(addTags), - addMetadata, - "active"), - CancellationToken.None); + var item = addCategory switch + { + "requirement" => await Memory.AddRequirementAsync( + new AddRequirementRequest(selectedProjectSlug, addRequirementStatement, addRequirementContext, ParseTags(addTags), "active"), + CancellationToken.None), + "location" => await Memory.AddLocationAsync( + new AddLocationRequest(selectedProjectSlug, addLocationItemName, addLocationPlace, addLocationDetails, ParseTags(addTags), "active"), + CancellationToken.None), + "todo" => await Memory.AddTodoAsync( + new AddTodoRequest(selectedProjectSlug, addTodoTask, addTodoPriority, ParseDueAt(addTodoDueAt), addTodoDetails, ParseTags(addTags), "active"), + CancellationToken.None), + _ => await Memory.AddNoteAsync( + new AddNoteRequest(selectedProjectSlug, addNoteSubject, addNoteBody, ParseTags(addTags), "active"), + CancellationToken.None) + }; message = $"Added {item.Title}."; ClearAddForm(); @@ -408,7 +556,7 @@ { searchResults = await Memory.FindItemsAsync( new FindMemoryItemsRequest( - searchQuery, + ParseSearchQueries(searchQueries), selectedProjectSlug, searchAllProjects, string.IsNullOrWhiteSpace(searchCategory) ? null : searchCategory, @@ -442,17 +590,21 @@ await GuardedAsync(async () => { - var updated = await Memory.UpdateItemAsync( - new UpdateMemoryItemRequest( - editing.ProjectSlug, - editing.Id, - editCategory, - editTitle, - editContent, - ParseTags(editTags), - editMetadata, - "active"), - CancellationToken.None); + var updated = editCategory switch + { + "requirement" => await Memory.UpdateRequirementAsync( + new UpdateRequirementRequest(editing.ProjectSlug, editing.Id, editRequirementStatement, editRequirementContext, ParseTags(editTags), "active"), + CancellationToken.None), + "location" => await Memory.UpdateLocationAsync( + new UpdateLocationRequest(editing.ProjectSlug, editing.Id, editLocationItemName, editLocationPlace, editLocationDetails, ParseTags(editTags), "active"), + CancellationToken.None), + "todo" => await Memory.UpdateTodoAsync( + new UpdateTodoRequest(editing.ProjectSlug, editing.Id, editTodoTask, editTodoPriority, ParseDueAt(editTodoDueAt), string.IsNullOrWhiteSpace(editTodoDueAt), editTodoDetails, ParseTags(editTags), "active"), + CancellationToken.None), + _ => await Memory.UpdateNoteAsync( + new UpdateNoteRequest(editing.ProjectSlug, editing.Id, editNoteSubject, editNoteBody, ParseTags(editTags), "active"), + CancellationToken.None) + }; message = $"Updated {updated.Title}."; editing = null; @@ -471,47 +623,118 @@ await LoadRecentAsync(); } - private void BeginEdit(SearchResultDto item) + private async Task BeginEditAsync(SearchResultDto item) { editing = new EditState(item.ProjectSlug, item.Id); editCategory = item.Category.ToString().ToLowerInvariant(); - editTitle = item.Title; - editContent = item.Content; - editTags = string.Join(", ", item.Tags); - editMetadata = item.MetadataJson; + LoadEditFields(item.Details, item.Tags); + await LoadEditEmbeddingsAsync(); } - private void BeginEdit(MemoryListItem item) + private async Task BeginEditAsync(MemoryListItem item) { editing = new EditState(item.ProjectSlug, item.Id); editCategory = item.Category.ToString().ToLowerInvariant(); - editTitle = item.Title; - editContent = item.Content; - editTags = string.Join(", ", item.Tags); - editMetadata = item.MetadataJson; + LoadEditFields(item.Details, item.Tags); + await LoadEditEmbeddingsAsync(); + } + + private void LoadEditFields(MemoryItemDetailsDto details, string[] tags) + { + editRequirementStatement = details.RequirementStatement ?? string.Empty; + editRequirementContext = details.RequirementContext ?? string.Empty; + editLocationItemName = details.LocationItemName ?? string.Empty; + editLocationPlace = details.LocationPlace ?? string.Empty; + editLocationDetails = details.LocationDetails ?? string.Empty; + editTodoTask = details.TodoTask ?? string.Empty; + editTodoPriority = string.IsNullOrWhiteSpace(details.TodoPriority) ? "normal" : details.TodoPriority; + editTodoDueAt = details.TodoDueAt?.ToString("O") ?? string.Empty; + editTodoDetails = details.TodoDetails ?? string.Empty; + editNoteSubject = details.NoteSubject ?? string.Empty; + editNoteBody = details.NoteBody ?? string.Empty; + editTags = string.Join(", ", tags); } private void CancelEdit() { editing = null; + editEmbeddings = []; + newEmbeddingLabel = string.Empty; + newEmbeddingText = string.Empty; } private void ClearAddForm() { - addCategory = "note"; - addTitle = string.Empty; - addContent = string.Empty; + addRequirementStatement = string.Empty; + addRequirementContext = string.Empty; + addLocationItemName = string.Empty; + addLocationPlace = string.Empty; + addLocationDetails = string.Empty; + addTodoTask = string.Empty; + addTodoPriority = "normal"; + addTodoDueAt = string.Empty; + addTodoDetails = string.Empty; + addNoteSubject = string.Empty; + addNoteBody = string.Empty; addTags = string.Empty; - addMetadata = """{"source":"web"}"""; } private void ClearSearch() { - searchQuery = string.Empty; + searchQueries = string.Empty; searchCategory = string.Empty; searchResults = []; } + private async Task LoadEditEmbeddingsAsync() + { + if (editing is null) + { + editEmbeddings = []; + return; + } + + editEmbeddings = await Memory.ListItemEmbeddingsAsync( + new ListItemEmbeddingsRequest(editing.ProjectSlug, editing.Id), + CancellationToken.None); + } + + private async Task AddEmbeddingAsync() + { + if (editing is null) + { + return; + } + + await GuardedAsync(async () => + { + _ = await Memory.AddItemEmbeddingAsync( + new AddItemEmbeddingRequest(editing.ProjectSlug, editing.Id, newEmbeddingLabel, newEmbeddingText), + CancellationToken.None); + newEmbeddingLabel = string.Empty; + newEmbeddingText = string.Empty; + await LoadEditEmbeddingsAsync(); + await LoadRecentAsync(); + }); + } + + private async Task DeleteEmbeddingAsync(Guid embeddingId) + { + if (editing is null) + { + return; + } + + await GuardedAsync(async () => + { + _ = await Memory.DeleteItemEmbeddingAsync( + new DeleteItemEmbeddingRequest(editing.ProjectSlug, editing.Id, embeddingId), + CancellationToken.None); + await LoadEditEmbeddingsAsync(); + await LoadRecentAsync(); + }); + } + private async Task GuardedAsync(Func work) { try @@ -539,6 +762,25 @@ .ToArray(); } + private static string[] ParseSearchQueries(string queries) + { + return queries + .Split(['\r', '\n'], StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries) + .ToArray(); + } + + private static DateTimeOffset? ParseDueAt(string dueAt) + { + if (string.IsNullOrWhiteSpace(dueAt)) + { + return null; + } + + return DateTimeOffset.TryParse(dueAt, out var parsed) + ? parsed + : throw new ArgumentException("Due must be a valid timestamp."); + } + private static MemoryCategory? ParseOptionalCategory(string category) { if (string.IsNullOrWhiteSpace(category)) @@ -549,5 +791,17 @@ return Enum.TryParse(category, ignoreCase: true, out var parsed) ? parsed : null; } + private static string DetailSummary(MemoryCategory category, MemoryItemDetailsDto details) + { + return category switch + { + MemoryCategory.Requirement => details.RequirementContext ?? details.RequirementStatement ?? string.Empty, + MemoryCategory.Location => $"{details.LocationItemName} -> {details.LocationPlace}".Trim(), + MemoryCategory.Todo => $"{details.TodoTask} [{details.TodoPriority}]".Trim(), + MemoryCategory.Note => details.NoteBody ?? details.NoteSubject ?? string.Empty, + _ => string.Empty + }; + } + private sealed record EditState(string ProjectSlug, Guid Id); } diff --git a/src/Cortex.Web/CortexUiQueries.cs b/src/Cortex.Web/CortexUiQueries.cs index 1b95d07..f46e8d8 100644 --- a/src/Cortex.Web/CortexUiQueries.cs +++ b/src/Cortex.Web/CortexUiQueries.cs @@ -1,4 +1,5 @@ using Cortex.Core.Data; +using Cortex.Core.Contracts; using Cortex.Core.Domain; using Microsoft.EntityFrameworkCore; @@ -47,22 +48,68 @@ public sealed class CortexUiQueries(CortexDbContext db) query = query.Where(item => item.Category == category); } - return await query + var items = await query .OrderByDescending(item => item.UpdatedAt) .Take(Math.Clamp(limit, 1, 100)) - .Select(item => new MemoryListItem( + .ToListAsync(cancellationToken); + var itemIds = items.Select(item => item.Id).ToArray(); + var embeddingSummaries = await db.MemoryItemEmbeddings + .Where(embedding => itemIds.Contains(embedding.MemoryItemId)) + .GroupBy(embedding => embedding.MemoryItemId) + .Select(group => new + { + ItemId = group.Key, + Count = group.Count(), + DefaultModel = group + .Where(embedding => embedding.Position == 0) + .Select(embedding => embedding.EmbeddingModel) + .SingleOrDefault() + }) + .ToDictionaryAsync(summary => summary.ItemId, cancellationToken); + + return items + .Select(item => + { + embeddingSummaries.TryGetValue(item.Id, out var embeddingSummary); + return new MemoryListItem( item.Id, item.Project!.Slug, item.Category, item.Title, item.Content, item.Tags, - item.MetadataJson, item.Status, + embeddingSummary?.DefaultModel, + embeddingSummary?.Count ?? 0, item.CreatedAt, item.UpdatedAt, - item.DeletedAt)) - .ToListAsync(cancellationToken); + item.DeletedAt, + DetailsFrom(item)); + }) + .ToList(); + } + + private static MemoryItemDetailsDto DetailsFrom(MemoryItem item) + { + return item switch + { + RequirementMemoryItem requirement => new MemoryItemDetailsDto( + RequirementStatement: requirement.Statement, + RequirementContext: requirement.Context), + LocationMemoryItem location => new MemoryItemDetailsDto( + LocationItemName: location.ItemName, + LocationPlace: location.Place, + LocationDetails: location.Details), + TodoMemoryItem todo => new MemoryItemDetailsDto( + TodoTask: todo.Task, + TodoPriority: todo.Priority, + TodoDueAt: todo.DueAt, + TodoDetails: todo.Details), + NoteMemoryItem note => new MemoryItemDetailsDto( + NoteSubject: note.Subject, + NoteBody: note.Body), + _ => new MemoryItemDetailsDto() + }; } } @@ -81,8 +128,10 @@ public sealed record MemoryListItem( string Title, string Content, string[] Tags, - string MetadataJson, string Status, + string? DefaultEmbeddingModel, + int EmbeddingCount, DateTimeOffset CreatedAt, DateTimeOffset UpdatedAt, - DateTimeOffset? DeletedAt); + DateTimeOffset? DeletedAt, + MemoryItemDetailsDto Details); diff --git a/tests/Cortex.IntegrationTests/Support/CortexDtos.cs b/tests/Cortex.IntegrationTests/Support/CortexDtos.cs index 461c3a5..f35b9c5 100644 --- a/tests/Cortex.IntegrationTests/Support/CortexDtos.cs +++ b/tests/Cortex.IntegrationTests/Support/CortexDtos.cs @@ -15,11 +15,12 @@ public sealed record MemoryItemDto( [property: JsonPropertyName("title")] string Title, [property: JsonPropertyName("content")] string Content, [property: JsonPropertyName("tags")] string[] Tags, - [property: JsonPropertyName("metadataJson")] string MetadataJson, [property: JsonPropertyName("status")] string Status, - [property: JsonPropertyName("embeddingModel")] string? EmbeddingModel, + [property: JsonPropertyName("defaultEmbeddingModel")] string? DefaultEmbeddingModel, + [property: JsonPropertyName("embeddingCount")] int EmbeddingCount, [property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt, - [property: JsonPropertyName("updatedAt")] DateTimeOffset UpdatedAt); + [property: JsonPropertyName("updatedAt")] DateTimeOffset UpdatedAt, + [property: JsonPropertyName("details")] MemoryItemDetailsDto Details); public sealed record SearchResultDto( [property: JsonPropertyName("id")] Guid Id, @@ -28,7 +29,33 @@ public sealed record SearchResultDto( [property: JsonPropertyName("title")] string Title, [property: JsonPropertyName("content")] string Content, [property: JsonPropertyName("tags")] string[] Tags, - [property: JsonPropertyName("metadataJson")] string MetadataJson, [property: JsonPropertyName("status")] string Status, [property: JsonPropertyName("score")] double Score, + [property: JsonPropertyName("matchedQueryIndex")] int MatchedQueryIndex, + [property: JsonPropertyName("matchedEmbeddingPosition")] int MatchedEmbeddingPosition, + [property: JsonPropertyName("matchedEmbeddingLabel")] string? MatchedEmbeddingLabel, + [property: JsonPropertyName("updatedAt")] DateTimeOffset UpdatedAt, + [property: JsonPropertyName("details")] MemoryItemDetailsDto Details); + +public sealed record MemoryItemEmbeddingDto( + [property: JsonPropertyName("id")] Guid Id, + [property: JsonPropertyName("memoryItemId")] Guid MemoryItemId, + [property: JsonPropertyName("position")] int Position, + [property: JsonPropertyName("label")] string? Label, + [property: JsonPropertyName("text")] string Text, + [property: JsonPropertyName("embeddingModel")] string? EmbeddingModel, + [property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt, [property: JsonPropertyName("updatedAt")] DateTimeOffset UpdatedAt); + +public sealed record MemoryItemDetailsDto( + [property: JsonPropertyName("requirementStatement")] string? RequirementStatement, + [property: JsonPropertyName("requirementContext")] string? RequirementContext, + [property: JsonPropertyName("locationItemName")] string? LocationItemName, + [property: JsonPropertyName("locationPlace")] string? LocationPlace, + [property: JsonPropertyName("locationDetails")] string? LocationDetails, + [property: JsonPropertyName("todoTask")] string? TodoTask, + [property: JsonPropertyName("todoPriority")] string? TodoPriority, + [property: JsonPropertyName("todoDueAt")] DateTimeOffset? TodoDueAt, + [property: JsonPropertyName("todoDetails")] string? TodoDetails, + [property: JsonPropertyName("noteSubject")] string? NoteSubject, + [property: JsonPropertyName("noteBody")] string? NoteBody); diff --git a/tests/Cortex.IntegrationTests/Support/McpTestClientExtensions.cs b/tests/Cortex.IntegrationTests/Support/McpTestClientExtensions.cs index 87067cc..f11624d 100644 --- a/tests/Cortex.IntegrationTests/Support/McpTestClientExtensions.cs +++ b/tests/Cortex.IntegrationTests/Support/McpTestClientExtensions.cs @@ -13,24 +13,80 @@ public static class McpTestClientExtensions }); } - public static ValueTask AddItemAsync( + public static ValueTask AddRequirementAsync( this McpClient client, string project, - string category, - string title, - string content, + string statement, + string context, string[]? tags = null, - string? metadataJson = null, string? status = null) { - return client.CallToolAsync("AddItem", new Dictionary + return client.CallToolAsync("AddRequirement", new Dictionary { ["project"] = project, - ["category"] = category, - ["title"] = title, - ["content"] = content, + ["statement"] = statement, + ["context"] = context, + ["tags"] = tags, + ["status"] = status + }); + } + + public static ValueTask AddLocationAsync( + this McpClient client, + string project, + string itemName, + string place, + string? details = null, + string[]? tags = null, + string? status = null) + { + return client.CallToolAsync("AddLocation", new Dictionary + { + ["project"] = project, + ["itemName"] = itemName, + ["place"] = place, + ["details"] = details, + ["tags"] = tags, + ["status"] = status + }); + } + + public static ValueTask AddTodoAsync( + this McpClient client, + string project, + string task, + string priority, + DateTimeOffset? dueAt = null, + string? details = null, + string[]? tags = null, + string? status = null) + { + return client.CallToolAsync("AddTodo", new Dictionary + { + ["project"] = project, + ["task"] = task, + ["priority"] = priority, + ["dueAt"] = dueAt, + ["details"] = details, + ["tags"] = tags, + ["status"] = status + }); + } + + public static ValueTask AddNoteAsync( + this McpClient client, + string project, + string subject, + string body, + string[]? tags = null, + string? status = null) + { + return client.CallToolAsync("AddNote", new Dictionary + { + ["project"] = project, + ["subject"] = subject, + ["body"] = body, ["tags"] = tags, - ["metadataJson"] = metadataJson, ["status"] = status }); } @@ -42,10 +98,21 @@ public static class McpTestClientExtensions bool searchAllProjects = false, string? category = null, int limit = 10) + { + return client.FindItemsAsync([query], project, searchAllProjects, category, limit); + } + + public static ValueTask FindItemsAsync( + this McpClient client, + string[] queries, + string? project = null, + bool searchAllProjects = false, + string? category = null, + int limit = 10) { return client.CallToolAsync("FindItems", new Dictionary { - ["query"] = query, + ["queries"] = queries, ["project"] = project, ["searchAllProjects"] = searchAllProjects, ["category"] = category, @@ -53,26 +120,70 @@ public static class McpTestClientExtensions }); } - public static ValueTask UpdateItemAsync( + public static ValueTask ListItemEmbeddingsAsync( + this McpClient client, + string project, + Guid itemId) + { + return client.CallToolAsync("ListItemEmbeddings", new Dictionary + { + ["project"] = project, + ["itemId"] = itemId + }); + } + + public static ValueTask AddItemEmbeddingAsync( + this McpClient client, + string project, + Guid itemId, + string text, + string? label = null) + { + return client.CallToolAsync("AddItemEmbedding", new Dictionary + { + ["project"] = project, + ["itemId"] = itemId, + ["text"] = text, + ["label"] = label + }); + } + + public static ValueTask DeleteItemEmbeddingAsync( + this McpClient client, + string project, + Guid itemId, + Guid embeddingId) + { + return client.CallToolAsync("DeleteItemEmbedding", new Dictionary + { + ["project"] = project, + ["itemId"] = itemId, + ["embeddingId"] = embeddingId + }); + } + + public static ValueTask UpdateTodoAsync( this McpClient client, string project, Guid id, - string? category = null, - string? title = null, - string? content = null, + string? task = null, + string? priority = null, + DateTimeOffset? dueAt = null, + bool clearDueAt = false, + string? details = null, string[]? tags = null, - string? metadataJson = null, string? status = null) { - return client.CallToolAsync("UpdateItem", new Dictionary + return client.CallToolAsync("UpdateTodo", new Dictionary { ["project"] = project, ["id"] = id, - ["category"] = category, - ["title"] = title, - ["content"] = content, + ["task"] = task, + ["priority"] = priority, + ["dueAt"] = dueAt, + ["clearDueAt"] = clearDueAt, + ["details"] = details, ["tags"] = tags, - ["metadataJson"] = metadataJson, ["status"] = status }); } diff --git a/tests/Cortex.IntegrationTests/Tests/MemoryItemTests.cs b/tests/Cortex.IntegrationTests/Tests/MemoryItemTests.cs index eb9d16f..563efc8 100644 --- a/tests/Cortex.IntegrationTests/Tests/MemoryItemTests.cs +++ b/tests/Cortex.IntegrationTests/Tests/MemoryItemTests.cs @@ -9,15 +9,15 @@ namespace Cortex.IntegrationTests.Tests; public sealed class MemoryItemTests(CortexTestFixture fixture) { [Fact] - public async Task Add_item_fails_when_project_is_missing() + public async Task Add_todo_fails_when_project_is_missing() { await using var client = await fixture.CreateClientAsync(); - var result = await client.AddItemAsync( + var result = await client.AddTodoAsync( fixture.NewProjectName("missing"), - "note", "Should not be stored", - "This item should fail because its project does not exist."); + "normal", + details: "This item should fail because its project does not exist."); Assert.True(result.IsError is true); } @@ -30,25 +30,35 @@ public sealed class MemoryItemTests(CortexTestFixture fixture) _ = await client.CreateProjectAsync(projectName); var added = CortexTestFixture.ReadStructuredContent( - await client.AddItemAsync( + await client.AddTodoAsync( projectName, - "todo", "Measure cabinet opening", - "Measure the kitchen cabinet opening before ordering shelves.", - ["kitchen", "measure"], - """{"priority":"normal"}""")); + "normal", + details: "Measure the kitchen cabinet opening before ordering shelves.", + tags: ["kitchen", "measure"], + status: "active")); Assert.Equal("Todo", added.Category); - Assert.Equal("BAAI/bge-small-en-v1.5", added.EmbeddingModel); + Assert.Equal("Measure cabinet opening", added.Details.TodoTask); + Assert.Equal("normal", added.Details.TodoPriority); + Assert.Equal("BAAI/bge-small-en-v1.5", added.DefaultEmbeddingModel); + Assert.Equal(1, added.EmbeddingCount); + + var defaultEmbeddings = CortexTestFixture.ReadStructuredContent( + await client.ListItemEmbeddingsAsync(projectName, added.Id)); + + Assert.Single(defaultEmbeddings); + Assert.Equal(0, defaultEmbeddings[0].Position); var updated = CortexTestFixture.ReadStructuredContent( - await client.UpdateItemAsync( + await client.UpdateTodoAsync( projectName, added.Id, - content: "Measure the kitchen cabinet opening and record width, height, and depth.", + details: "Measure the kitchen cabinet opening and record width, height, and depth.", status: "active")); Assert.Contains("width", updated.Content); + Assert.Contains("width", updated.Details.TodoDetails); Assert.True(updated.UpdatedAt >= added.UpdatedAt); var deleted = CortexTestFixture.ReadStructuredContent( @@ -61,4 +71,37 @@ public sealed class MemoryItemTests(CortexTestFixture fixture) Assert.DoesNotContain(searchAfterDelete, item => item.Id == added.Id); } + + [Fact] + public async Task Additional_embedding_can_match_search_without_changing_content() + { + await using var client = await fixture.CreateClientAsync(); + var projectName = fixture.NewProjectName("embeddings"); + _ = await client.CreateProjectAsync(projectName); + + var note = CortexTestFixture.ReadStructuredContent( + await client.AddNoteAsync( + projectName, + "Build diagnostics", + "Keep the diagnostics concise and focused on failing subsystems.")); + + var extra = CortexTestFixture.ReadStructuredContent( + await client.AddItemEmbeddingAsync( + projectName, + note.Id, + "Mentions observability, telemetry, traces, and runtime health checks.", + "observability")); + + Assert.Equal(1, extra.Position); + + var results = CortexTestFixture.ReadStructuredContent( + await client.FindItemsAsync(["telemetry traces", "runtime health"], projectName)); + + var match = Assert.Single(results, result => result.Id == note.Id); + Assert.Equal(1, match.MatchedEmbeddingPosition); + Assert.Equal("observability", match.MatchedEmbeddingLabel); + + var deleteDefault = await client.DeleteItemEmbeddingAsync(projectName, note.Id, note.Id); + Assert.True(deleteDefault.IsError is true); + } } diff --git a/tests/Cortex.IntegrationTests/Tests/SearchTests.cs b/tests/Cortex.IntegrationTests/Tests/SearchTests.cs index 6463112..fd0f89b 100644 --- a/tests/Cortex.IntegrationTests/Tests/SearchTests.cs +++ b/tests/Cortex.IntegrationTests/Tests/SearchTests.cs @@ -19,9 +19,9 @@ public sealed class SearchTests(CortexTestFixture fixture) _ = await client.CreateProjectAsync(beta); var alphaItem = CortexTestFixture.ReadStructuredContent( - await client.AddItemAsync(alpha, "location", "Charging brick", "The USB-C charging brick is in the blue backpack.")); + await client.AddLocationAsync(alpha, "USB-C charging brick", "blue backpack")); - _ = await client.AddItemAsync(beta, "location", "Charging brick", "The spare USB-C charging brick is in the garage drawer."); + _ = await client.AddLocationAsync(beta, "spare USB-C charging brick", "garage drawer"); var alphaResults = CortexTestFixture.ReadStructuredContent( await client.FindItemsAsync("where is the charging brick", alpha)); @@ -39,9 +39,9 @@ public sealed class SearchTests(CortexTestFixture fixture) _ = await client.CreateProjectAsync(projectName); var note = CortexTestFixture.ReadStructuredContent( - await client.AddItemAsync(projectName, "note", "Shelf idea", "Use oak shelves for the reading nook.")); + await client.AddNoteAsync(projectName, "Shelf idea", "Use oak shelves for the reading nook.")); - _ = await client.AddItemAsync(projectName, "todo", "Buy shelf brackets", "Buy black metal brackets for the oak shelves."); + _ = await client.AddTodoAsync(projectName, "Buy shelf brackets", "normal", details: "Buy black metal brackets for the oak shelves."); var allProjects = CortexTestFixture.ReadStructuredContent( await client.FindItemsAsync("oak shelves reading nook", searchAllProjects: true, category: "note")); diff --git a/tests/Cortex.IntegrationTests/Tests/SmokeTests.cs b/tests/Cortex.IntegrationTests/Tests/SmokeTests.cs index 250e84b..a89736d 100644 --- a/tests/Cortex.IntegrationTests/Tests/SmokeTests.cs +++ b/tests/Cortex.IntegrationTests/Tests/SmokeTests.cs @@ -26,9 +26,19 @@ public sealed class SmokeTests(CortexTestFixture fixture) Assert.Contains("CreateProject", names); Assert.Contains("ListProjects", names); - Assert.Contains("AddItem", names); + Assert.Contains("AddRequirement", names); + Assert.Contains("AddLocation", names); + Assert.Contains("AddTodo", names); + Assert.Contains("AddNote", names); Assert.Contains("FindItems", names); - Assert.Contains("UpdateItem", names); + Assert.Contains("ListItemEmbeddings", names); + Assert.Contains("AddItemEmbedding", names); + Assert.Contains("UpdateItemEmbedding", names); + Assert.Contains("DeleteItemEmbedding", names); + Assert.Contains("UpdateRequirement", names); + Assert.Contains("UpdateLocation", names); + Assert.Contains("UpdateTodo", names); + Assert.Contains("UpdateNote", names); Assert.Contains("DeleteItem", names); } }