Building the product
The tools we built to build DataTug.
You should be able to trust the software you use to understand your data. For us, that trust starts with how we build: on proven open-source foundations, with explicit contracts, reviewable changes, and evidence that a feature reached the product. The tools we created for that work are open for other developers too.
Trust is built during development
Suppose we want DataTug to let a developer attach a shared meaning to a query result column. That small sentence crosses the query editor, a saved project definition, typed result metadata, a semantic registry, a local service, and a hosted service. It is a useful example because the engineering problem is familiar: each part is reasonable on its own, but the contract between them can disappear into tickets, chat, and memory.
Our starting point is not a new platform for every problem. Git gives us history and review. Markdown keeps decisions readable. Go and TypeScript let us build practical services and interfaces. SQLite gives local tools a dependable store. We add our own formats and tools when a connection between those foundations needs to be explicit, portable, and testable.
Turn the request into a checkable promise
“Let people add meanings to columns” is an idea, not yet a build contract. Which columns are eligible? What happens if the result shape changes? Does an unconfirmed mapping look valid? SpecScore is our open format and methodology for recording features, requirements, acceptance criteria, plans, and tasks as Markdown in Git. Its CLI checks structure and references, so the decision can travel with the implementation and survive an agent handoff.
An acceptance criterion can state that a mapping awaiting validation has a neutral status, while a confirmed fingerprint mismatch needs review. A test and a reviewer can then check the same promise. Git and Markdown remain the foundation; SpecScore adds a traceable, lintable bridge from intent to code.
Find the implementation seam before editing
Now the developer needs to discover where result columns are constructed, where saved queries are written, and which screens consume them. Search is a great first step, but matching a symbol name does not show its callers or the path from a route to a database write. CodeGrapher indexes files, symbols, and relationships into a local SQLite graph that people and agents can query.
Trace the result-column type from the executor into the browser grid, then find the save path and the code that reads it back. The graph makes an impact review faster and makes omissions easier to spot. Our Go port builds on Colby McHenry's codegraph and uses a patched fork of Oscar Villavicencio's gotreesitter. That lineage is an example of taking useful open work further and publishing our changes.
CodeGrapher websiteSource repositoryOriginal codegraphOriginal gotreesitter
Make the data contracts say what they mean
A result column has a structural type and a business meaning. They are related, but they are not interchangeable. ModelSpec describes record types, fields, keys, and references independently of one database, language, or API framework. MeaningGraph defines reusable concepts with versioned addresses and lets a dataset bind its fields to them. DataTug owns the binding between a particular query result and those definitions.
The result field may be a decimal in the model while its meaning is “invoice total” in a shared concept graph. A calculated result can be mapped at query-result scope without pretending it came from a physical source column. We created separate standards because SQL DDL or generated types can describe shape but cannot settle semantic identity; a glossary can describe meaning but cannot validate a field's type or key.
ModelSpec websiteModelSpec sourceMeaningGraph websiteConcepts and format
Keep the query definition portable
Changing metadata on a saved query should not lock the feature to a single SQL dialect or make the browser and CLI maintain different query models. TugQL is the open query language we are developing for relational, document, API, and graph-connected sources. Its structured tree, TugQT, gives editors and agents a common representation to inspect and transform. Older DataTug libraries and documents still use the former name DTQL.
A saved query can keep its readable text while a mapping refers to its output field and the definition revision. We built a portable query contract because raw SQL is tied to a provider and a visual builder's private JSON would be hard to review in Git. Provider-specific execution remains the job of an adapter, with its limits made explicit.
Implement once, choose storage deliberately
The feature needs to save and read data in several settings. DALgo gives Go code a shared API for records, queries, transactions, and access policies across adapters. It does not erase provider differences: an adapter reports a capability it cannot support. We built DALgo because wiring business rules directly to Firestore, SQL, or a test store would duplicate the same logic and make backend changes expensive.
Some definitions should live in the project repository. inGitDB treats structured records as schema-validated files in Git, so a change can be diffed and reviewed with code. We made it because an operational database is excellent at serving requests but does not naturally make each definition a branchable, human-readable pull request.
Private operational state needs another boundary. OpenVaultDB gives databases an identity and mounted engines under a common contract. DataTug is building its session storage on that boundary with local SQLite and hosted Firestore profiles. Its broader user-owned vault and grants vision is still evolving. The session storage article has the detailed architecture and current limits.
DALgo is the application code's data access seam. inGitDB is for structured content that belongs in version control. OpenVaultDB is the database and ownership boundary for application state. They compose where useful, but none is a rename of another.
DALgo websiteDALgo sourceinGitDB websiteinGitDB CLI sourceOpenVaultDB websiteOpenVaultDB server source
Test the conversation a developer actually has
A conversational feature needs checks across several turns: what the user sends, what the application replies, and what happens after the next action. Chatwright is our open-source tool for developing and testing conversational applications. It emulates messaging-platform APIs, drives the real application through HTTP, and records interactions in a run bundle developers can inspect in a player.
We created it because a unit test around a reply function leaves out webhook delivery, button actions, message edits, and the sequence a user experiences. Established messaging APIs and Go's testing tools provide the foundations. Chatwright adds repeatable conversation scenarios and evidence at that boundary, so we can investigate a failure without manually repeating the conversation on a live messaging service.
Chatwright's greeting bot receives /start, offers language buttons, and edits its reply when a user selects another language. The test checks the interaction through the emulated platform. Its runtime also has an agent CLI example that asks an agent to remember a word, then checks recall on the next turn. That is the kind of continuity a conversational data tool must get right before a longer workflow can be trusted.
For DataTug, the proposed integration would let developers inspect captured data assertions and query results in its workspace. That handoff remains design work. Chatwright's public examples already let developers exercise the conversation-testing mechanism independently.
Chatwright website and playerChatwright specification and formatsGo runtime and agent examplesRunnable greeting-bot example
Prove the change reached the product
A passing unit test in one checkout is useful evidence, but this feature can cross the DataTug app, CLI, query library, and registry. WB, the Sneat Dev Workbench, coordinates repository work: it creates isolated worktrees, runs repeatable checks, helps land compatible changes, and records which exact remote target received them. Review, CI, and deployment checks remain separate steps. That distinction matters when someone asks whether a feature is actually available.
The developer can validate the changed query and mapping behavior, review the site copy against what actually works, and land each repository with its own CI receipt. WB adds a verifiable lifecycle around Git and CI, so a local green test is never mistaken for a remote release.
Built on open source. Shared back as open source.
Each new tool is maintenance work, so we build only where a shared contract is missing: a specification to lint, a code path to trace, a model to reuse, a meaning to pin, a query to move, or a change to land with evidence. We publish the formats and reference implementations with their source, examples, and limitations. Other teams can inspect the decisions, use a tool independently of DataTug, file an issue, or build a different implementation of an open format.
Some parts are in routine use and others, especially the wider semantic registry and user-owned vault experience, are still under development. We say which is which because a clean specification, a merged library, and a deployed product are different kinds of evidence. For developers evaluating DataTug, that is the promise behind this article: our process should make the software understandable, its claims checkable, and its building blocks useful beyond our own product.