Guide to data prep and processing
What you will learn in this module
Four key benefits of data federation
How AI has altered the requirements of data prep
Why you should apply data governance at the data layer
Two ways AI can help you prepare data for AI consumption
Data prep and processing
Our survey of over 200 data and analytics leaders found that only 7% of enterprises have moved past AI pilots and into organization-wide adoption. As we’ve discussed, one of the biggest hurdles is trust — and more specifically, what it takes to get data into a decision-ready state.
The data prep and processing plumbing most teams deploy today was scoped for a much more specific task. The business team brings you a specific set of questions, and you build a pipeline and model for each one.
AI is changing that mindset. Today, every business function wants to ask and answer their own data questions. But no data team can effectively prep data at the speed people ask questions. That’s why, the way we think about data pipelines also has to change.
In this new model, your job shifts from building pipelines for scoped questions to building the infrastructure for AI to answer any question, both accurately and consistently. This lesson covers the four decisions that follow: what to move, what to leave in place, what shape to give it, and who is allowed to reach it.
Data federation: query the data where it lives
Data federation lets you query live data across multiple sources without copying it into a central store.
As a refresher from the last section:
A context layer sits above your warehouses, databases, SaaS applications, APIs, and documents. It’s the central repository for definitions, business rules and procedures, data models and their descriptions, the ontology that relates business entities to each other, access and data policies, and the institutional knowledge that usually lives in one analyst’s head.
The analytics harness accepts a query, uses the context layer to correctly interpret it, and works out the most efficient way to extract the necessary data from each source. It then combines and verifies that answer, before sending it back to the insight delivery surface.
Pushdown makes this possible. The analytics harness translates each sub-query into what the source expects and pushes filtering and aggregation down to the source itself. As an added benefit, the results that travel back are already reduced and optimized. The harness’s ability to push down depends on what each source connector supports. Each unsupported filter or join still has to be processed centrally.
Four things that change when you federate your sources
Faster time to value: No pipeline build, no landing zone, no data engineering queue. Simply connect your data and start answering questions in days, instead of quarters.
Freshness: Queries run against source systems directly, so an agent can work with current data rather than the state of the last ingestion cycle.
Structured and unstructured together: One layer can reach your warehouse tables alongside the contracts, tickets, and transcripts that explain them.
Fewer copies to maintain: You replicate less data and maintain fewer sync jobs. But watch the other side of the ledger, because queries that hit live sources add compute and network load there. That’s why optimized push downs are important.
Rather than replacing your existing pipelines, you should think of federation as a complement. It is especially valuable when insights need to span distributed, live data without first centralizing or duplicating.
Evolving data transformation for AI analytics
Classic transformation practices work to reshape data for human readability: you receive a list of metrics from the business, and you build a pipeline and deliver the answer through a dashboard or report. To deliver, you pre-join, pre-aggregate, and flatten source detail into the columns a dashboard needs.
However, that methodology discards necessary context that an agent may need to accurately answer natural-language, self-serve business questions. To prepare your data for AI, your transformation practices need to evolve.
Key process changes to look out for:
Schema ingestion: This process exposes database tables, foreign keys, and data types directly at the source, so an agent can see the structural topology it has to reason over: i.e., an order is always linked to an account, a status column is an enumeration. While cleaning, deduplication, and standardization still matter in AI analytics, it’s important to understand its limitations. Notably, schema ingestion tells the agent what the shape is, but it only shares what the schema declares. When your warehouse omits foreign keys or stores a business enumeration as free text, you must supply that relationship as metadata instead.
Unstructured sources: This data type requires the same level of special care, but in a different form. A PDF or a Word document should be modeled on its structural boundaries — headings, sections, and logical units — rather than splitting every fixed number of characters. A split made on character count can cut a definition in half and retrieve a fragment that reads fluently and means nothing. Attaching the surrounding context to each unit before you index it improves retrieval on the corpora. Anthropic tested it.
Semantic modeling: Domain specific metrics and business glossaries are traditionally tracked through a semantic layer. Any existing semantic can be ingested into the context layer. Or, if you’re starting from scratch, you can build them natively in a context layer. Be sure you define ARR, CSAT, or net revenue retention once, as a governed calculation with its own rules for discounts, delinquency, and currency. That governed definition is what makes the organization’s version of a metric available to an agent. However, supplying the definition does not, by itself, force the agent to use it. That’s why you must validate the queries an LLM generates against those definitions.
Normalization: One more property is easy to lose: how a representation carries numbers and relationships. When you flatten structured data into loose text fragments, like AI skills, an agent can retrieve a sentence describing last quarter’s sales. However, it loses the ability to filter, join, or recompute it. By ensuring agents maintain the ability to reason over governed dataframes, with types and keys carried alongside, the numbers remain computable and the relationships stay intact.
Key takeaway
Modeling for agentic consumption is different from modeling for a dashboard. Agentic modeling should preserve structure, definitions, and precision. Anything you omit has to be reconstructed from source before an agent can use it for unscoped questions.
Governance and security at the data layer
Start with what you are legally required to do. Your obligations vary by company, industry, and country, and mismanagement carries civil and sometimes criminal consequences. The EU’s GDPR sets principles for lawfulness, purpose limitation, and data minimization that apply whether or not an AI system is involved, and the NIST AI Risk Management Framework gives you a structure for the AI-specific risks on top.
Next, check your own perimeter. Your organization’s security posture may prohibit exposing certain data to a model at all or to a model running in a particular environment. Treat that as a design input rather than an obstacle to work around.
Your deployment architecture decides what actually leaves your environment. For instance, WisdomAI describes an arrangement where data, credentials, and query execution stay inside the customer’s instance and the answer is what crosses into the model’s boundary. Whatever platform you run, trace it for yourself: which component holds the credentials, what the model receives in its prompt, where tools execute, and what the response itself contains.
Day to day, we recommend applying the principle of least privilege. Most of this work is contained within setting up the presentational context and row-level or column-level security.
Row and column policies decide which records and which fields a role may see, and the data platform enforces them under the identity of the requesting user.
Presentational context is a separate capability layered on top: it shapes how a scoped answer is delivered and explained.
Map both into your context layer, and one question will only return one correctly scoped answer per role. Your CFO sees the financial details. A business manager sees their own scope, and they can tell why it is scoped that way.
Go deeper
Read WisdomAI’s context product overview for how a governed context layer sits above existing data and semantic systems.
Where AI can help
At face value, prepping for any question may seem like a much more daunting task than prepping data for a clearly defined set of questions. But in reality, this systems-thinking approach saves you more time in the long run. Even better, AI can help. Here’s how:
Initial build
Predicting join paths and mapping table relationships is the manual part of connecting a new source, and a system that proposes them from metadata and query history removes most of that keyboard time.
WisdomAI bootstraps this context from assets you already have, including dbt projects, BI models, data catalogs, and query history — collapsing months of data plumbing into days of configuration; AI-augmented relationship discovery is one part of onboarding a source. You can measure the time savings against your own baseline.
Whenever working with AI, it’s important to treat every proposal as a candidate, not a truth. The domain owner should always decide which join is authoritative, no matter how confident AI’s suggestion is phrased.
Upkeep
Schemas change, columns get renamed, and sources come and go. By incorporating an AI-augmented feedback loop to monitor changes and suggest updates, AI can help you keep your context layer current. Specifically, AI can mine query histories and correlate user corrections, providing suggested optimizations for a domain owner to confirm.
WisdomAI’s continuous learning loop works thanks to the Analytics Harness pulling feedback into the Enterprise Context Layer. Our Context builder let’s you govern the way context changes over time, with expert review and approval that allows you to version, publish, and rollback.
Where this leads
Federating the sources, shaping them for a machine, and governing them at the data layer makes your data accessible, current, and permitted. It does not make an answer correct. An agent can reach every system in your company and still return a confident number that answers a different question than the one you asked. The next module covers what goes wrong, even when the plumbing is sound.
Key takeaways
Federation allows you to query and update live data where it sits, which gives an agent fresher data without replacing your pipelines.
Preparing data for AI preserves structural topology, models unstructured sources on their own boundaries, and keeps metrics in a governed semantic layer.
Key governance concern: know your regulatory obligations, trace what actually leaves your environment, and map RBAC, guardrails, and row- and column-level rules.
AI can augment data and context modeling, but a dedicated owner still needs to confirm what becomes authoritative. That owner is you, the AI Context Engineer.