The whole surface,part by part
DjiniousData is one Bun process — HTTP, WebSocket and background jobs — over SurrealDB for records, graph, vectors and full text, and TimescaleDB for time series, with a React client on top. This page walks the surface: what loads data, what models it, what resolves it, what watches it, and what an AI agent is and is not allowed to do with it.
Ingest
The document store
Upload into persistent folders, select what you want, and ingest. CSV, XLS/XLSX, PDF, DOCX, JSON, Markdown and plain text are parsed in-process — PDF and DOCX without an external binary — and each tabular row becomes its own knowledge object.
Connectors
External systems arrive through connector instances, each stamping its own source name so a dashboard can filter to exactly one feed. Connectors carry a sync interval, and a recurring scheduler job syncs the ones whose interval has elapsed.
The pipeline
Seven stages, in order: parse, normalize, deduplicate, permission-map, enrich, embed, persist. Each is a pure function of the envelope it is handed, which is why the same chain runs for a file upload, a connector sync and a fixture seed.
Provenance, validated
Every object records its source system, its source object id, its original URI, its ingestion time and its pipeline version. An object that fails provenance validation raises a terminal error instead of being stored — and a derived object with no parent references fails too.
Deduplication
A unique index on (source, external id) plus a content hash. Re-ingesting an unchanged row touches its update time and nothing else; a changed row is updated in place. Two writers racing for the same key converge on one record rather than two.
Time series alongside
Numeric columns are also written to a TimescaleDB metrics table on the same load, so a threshold rule and a trend chart read the same ingest rather than a second, drifting copy of it.
Model
Types are records
Entity types and relation types are rows, created through the UI or the API. Fields, labels, plurals, icons and a status workflow are all data. There is no generated code and no migration between defining a type and using it.
Generic CRUD
All entity reads and writes go through one set of endpoints under `/api/entities`, driven by the type definition. A new type gets its list view, detail page, search facets and agent tools without a line being written for it.
Per-organization overlays
An overlay keyed by (type, organization) layers extra fields, labels and workflow states over a shared base type, resolved for the caller's own organization. One tenant's extension does not become every tenant's schema.
Entity ↔ organization
Every entity carries a derived relation to the organization that owns it. It is synthesised from the ownership field rather than stored as an edge, so it cannot fall out of step with the record, and it is read-only in the graph explorer.
AI-proposed types
The assistant can read what has been ingested and propose an ontology for it — types, fields, relations. The proposal is a record you review and accept or reject; it is never applied on the model's own authority.
Resolve
Deterministic matching
A shared identifier scores 1.0, an exact normalised name 0.85, a fuzzy name 0.7. The matchers are pure functions with no model behind them, so the same two records score the same today and next quarter.
Clustering, and the line
Blocking, pairwise scoring, then union-find. Pairs at or above 0.85 are merged automatically; a fuzzy 0.7 stays a candidate. That gap is deliberate — it is where a person belongs.
Analyst decisions are durable
When an analyst confirms or rejects a candidate the decision is stored, and the next resolution run honours it. Re-running the pipeline does not re-litigate a judgement someone already made.
Edges carry evidence
Materialized graph edges keep references to the objects they were derived from, so an edge answers 'why do you think these are related' with rows rather than with confidence.
Scoped traversal
Neighbour lookups filter by the caller's permissions before returning, not after. A graph that filtered afterwards would still have told you the node existed.
Watch
Widgets over records
Metric, breakdown, trend and map widgets query the knowledge objects directly, with a facet, a filter, a metric field and an aggregation. There is no materialized dashboard table to refresh, and therefore none to be stale.
The map widget
The latest position per group, plotted on the same map component the rest of the app uses. It is fed by the ordinary ingest, so a fleet appears on a dashboard because its telemetry was loaded, not because a mapping feature was bought.
Telemetry alert rules
A rule watches a max, min, average or count of a whitelisted field over one source, grouped — per locomotive, per site, per device — and fires when it crosses a threshold. Rules are owner-scoped and evaluate against what that owner can see.
Shared dashboards
A dashboard is a named record scoped to an organization and a workspace. Publishing one is how a team gets the same view, rather than everyone rebuilding it from the same widget catalogue.
Live updates
Change streams from the database fan out through a WebSocket hub, so a page reflects a write from another session without a poll loop underneath it.
Vision
Detection as ingest
An object-detection pipeline runs over a stored document, a public URL or a live stream, and its results materialize three ways: metrics for charting and alerting, a knowledge object for search, and optionally an entity per tracked object.
Detections are not objects
A tracker collapses hundreds of boxes of one parked truck into one track, because the number that belongs on a dashboard is 'one truck', not 'nine hundred detections'.
The sampling caveat, stated
Tracking assumes consecutive sampled frames overlap. On 1080p60 footage of a single walking worker, a stride of 30 reported 13 distinct people and a stride of 5 reported 2. Detection counts are unaffected; only the unique-object count degrades. Keep the sampled interval under ~0.2 s when that number matters.
Streams are opt-in
Cameras are private-network by nature, so a stream source is refused unless its prefix is on the operator's allow-list. Public URLs are checked against the same guard the platform uses for webhooks.
Partial results survive
The worker posts detections back incrementally, which is what makes a live camera watchable and what preserves the work when a long run dies halfway.
Act
Signals
Analysis produces signals — anomalies, patterns, weak signals — each with a subject, a method, a confidence and references to the objects behind it. Severity is derived from confidence rather than chosen.
Investigations
A case collects signals, notes and evidence items into one thread with an owner and a priority, so a finding becomes work rather than a card that scrolls away.
Reports
Structured reports with typed sections and citations back to the source objects. A report is generated from the platform's own state, so a figure in a report and the tile it came from cannot disagree.
Automations
Scripts with schedules and webhooks, versioned, with run logs. They call the platform's own API, which means an automation is subject to the same permissions as the person who wrote it.
Export
PDF and Excel export from the same records the screen is reading, plus a documented REST surface for everything else. The API is not a subset of the UI; the UI is a consumer of the API.
Two stores, and the reason there are two
Everything above rests on a split worth stating plainly: the platform's records, graph, vectors and full-text index all live in one SurrealDB database, and only time series live in PostgreSQL.
- SurrealDB owns
- Users, credentials, sessions, organizations, workspaces and memberships; the ontology and its overlays; knowledge objects with their embeddings and full-text index; resolved entities and graph edges; dashboards, alert rules, signals, investigations and reports; the audit log and the job queue.
- TimescaleDB owns
- The metrics hypertable — the numeric series pulled out of ingested rows and posted by the vision worker, where a time-bucketed query over millions of points is the whole job.
- Why split at all
- One store for records means the graph, the vector search and the full-text search all see the same rows with the same permissions, with no synchronisation between them. Time series is the one workload that genuinely wants a different engine, so it gets one — and nothing else does.
- The cost, named
- Two connections, two failure modes, and metrics that can be behind the records they were derived from. The ingest writes both from the same pass so the window is small, but it is not zero and it is not hidden.
- Identity
- A JWT signed with HMAC-SHA256 over an Argon2id password hash, with credentials federated to a shared identity database so one account works across every Djinious app. Scoped API tokens exist for agents and are separate principals, not shared logins.
Walk the surface with us
A demo goes through this page in the running product — ingest a file, model what came out of it, resolve it, and put an alert on it inside the hour.