Security, Privacy and AI Governance

After this lesson, you will be able to prepare a customer security review, draw the data flows and trust boundaries around an AI feature, and identify which questions need an owner from security, privacy, legal, or platform engineering. The running NorthPort Freight example is fictional. It contains synthetic data only.

A customer asks, "Where does our dispatch data go, who can see it, and what happens if the assistant is manipulated?" That is a security and privacy review question, not a request for a confident yes-or-no answer from the FDE. Your job is to describe the system accurately, identify the decision owners, gather evidence, and track the questions that still need approval.

Security reduces the chance and impact of unauthorized access, misuse, disruption, or change. Privacy concerns how information about people is collected, used, shared, retained, and deleted. They overlap, but neither replaces the other. A system can encrypt information and still collect more personal data than the task needs. It can also minimize data and still allow a user to retrieve another customer's record because an authorization check is missing.

The FDE does not determine whether a customer's system complies with every law or standard. Legal obligations depend on the system's purpose, data, location, organizational roles, and other facts. Bring qualified security, privacy, and legal owners into the review. This lesson gives an engineering workflow and awareness-level pointers, not legal advice or a compliance certification.

Start the security review with a system map

Before completing a questionnaire, describe what is being built and how it will be used. Name the user, task, system owner, environment, external services, data sources, outputs, and actions the system can take. Include the AI model, retrieval index, API gateway, customer systems, logs, support tools, and human review step. A drawing helps reviewers spot an integration or data transfer that a feature description leaves out.

A trust boundary is a point where data or control moves between components or parties with different permissions or security assumptions. For the NorthPort assistant, boundaries include a dispatcher browser to the customer API, the API to a model service, and the application to the dispatch system. A path crossing a boundary needs an owner, an authentication method, an authorization rule, and an explanation of what data crosses it. Do not draw only the happy path; include logs, backups, analytics, evaluation exports, and support access where they apply.

Write down data categories rather than copying a real customer record into the design document. The assistant might receive an exception code, a route identifier, a dispatcher question, and approved procedure passages. Ask whether a route identifier can be linked to a driver or customer. A field that looks like a harmless ID can still identify a person when joined with another system. Identify personal data, confidential business data, credentials, and data the feature does not need.

Data minimization means limiting collection and use to what is needed for the stated purpose. For NorthPort, the model may need the exception type and the relevant procedure text, but not a driver's personal phone number. Confirm what the model provider receives, whether prompts or outputs are retained, whether they are used for training, where processing occurs, how long logs persist, and whether provider staff or subprocessors may access them. These are questions for the actual service terms and customer agreement. Do not assume that a service has the same data controls in every product tier or region.

Data residency often refers to where data is stored or processed, but a customer's requirement may also cover backups, logs, support access, disaster recovery, and onward transfers. Ask the customer what their term means and which contract or policy defines it. A cloud region setting alone may not answer every question about data access or processing. The cloud, model provider, monitoring vendor, and support workflow can each introduce another location or recipient.

Control access to APIs and tools

Authentication establishes which user, service, or system is making a request. Authorization decides what that authenticated identity may do to a specific resource. A valid login does not imply permission to read every dispatch record. Check authorization at the operation and object level: the API should confirm that this dispatcher is allowed to view this shipment or exception, rather than trusting an object ID supplied by the browser.

Use the customer's identity provider and approved authentication pattern. For services that use OAuth 2.0, follow the customer's current implementation guidance and relevant best practices. IETF RFC 9700 is the OAuth 2.0 Security Best Current Practice; it discusses threats such as token leakage and recommends secure authorization flows. This is a protocol reference, not a substitute for reviewing a specific implementation. Keep tokens out of URLs, logs, prompts, screenshots, and tickets. Store service credentials in an approved secret manager, scope them to the smallest needed permissions, and rotate them through the customer's process.

For APIs, define the actor, resource, permitted action, and expected response for every route. Test that a dispatcher cannot change a delivery they do not own, request administrative operations, or receive fields beyond their role. OWASP's API Security Top 10 highlights object-level authorization, authentication, object-property authorization, and unrestricted resource consumption among API risks. Those categories are useful review prompts. They are not a complete penetration test.

An AI agent may also call tools. Give each tool one narrow purpose, validate its arguments on the server, and enforce authorization in the tool implementation. A prompt instruction such as "only update the caller's own shipment" is not an access control. Treat retrieved text and user-provided content as untrusted input. A document can contain instructions that try to redirect an agent or expose data. The AI Agents lessons on prompt injection and tool sandboxing and privacy, bias, and red teaming, plus Prompt Engineering's safe prompting and prompt injection, cover those attack patterns in more depth.

Limit what an agent can do when an action has a real effect. Use read-only access for lookups where possible. Require a customer-approved confirmation for writes that affect dispatch operations. Set rate limits and resource bounds so a loop or abusive caller cannot consume unlimited capacity. Log actions with a request identifier and actor identity while minimizing the content stored in that audit record. Decide how emergency access is granted and removed.

Build security into the delivery work

DevSecOps is the practice of including security in development and operations work throughout the delivery lifecycle. It is not a scanner that runs at the end. NIST's Secure Software Development Framework (SSDF) describes high-level security practices that can be integrated into an organization's chosen software development process. For an FDE engagement, the practical application is to find security requirements early, review dependencies and configuration, test access boundaries, handle vulnerabilities through the customer's process, and prepare an operational response before release.

Work with the customer to understand their threat model. A threat model is a structured description of what can go wrong, who or what could cause it, and which controls reduce the likelihood or impact. Ask what the assistant can access, what an attacker could influence, what a compromised credential could reach, and what happens if a model endpoint is unavailable. Consider prompt injection, cross-tenant access, unsafe tool calls, exposed secrets, malicious or stale source content, denial of service, and accidental data leakage. Record mitigations and owners. Do not promise that a single test eliminates a class of risk.

Use the customer's approved dependency and image scanning tools. Review findings with the security owner because a scanner can report issues that do not apply or miss issues that do. Pin dependencies and deployment artifacts according to customer policy. Keep secrets out of source control and build artifacts. Separate development, staging, and production credentials. Make security changes through code review and the customer's change process, and preserve evidence of the reviewed version.

Define what will be logged before enabling verbose traces. Useful security events can include sign-in outcomes, authorization denials, tool invocations, configuration changes, and administrative access. Prompts, model outputs, tool arguments, and retrieved passages can contain confidential or personal data. Set retention and access based on the purpose and customer policy. The debugging lesson on customer-safe observability explains how to investigate with a minimal, redacted record.

Agree on data processing and suppliers

A data processing agreement (DPA) is a contract or legal instrument that describes how one party processes data for another and the obligations that apply. The parties' roles depend on what they actually decide and do, not only on a label in a template. In the EU GDPR context, the European Commission describes a controller as the party that determines the purposes and means of processing, and a processor as one processing on the controller's behalf. The Commission also describes a binding contract for processor duties and prior written authorization for subprocessors. Other jurisdictions and agreements can use different terms or requirements.

Before sending customer data to a model or hosting service, identify the customer-approved product and service terms. Confirm the processing purpose, data categories, region, retention, deletion, security measures, subprocessors, breach notification path, support access, and whether data can be used to improve a provider's services. Check whether a transfer to another country or vendor requires a separate approval or contract. Do not infer the answer from a public product page if the customer's negotiated terms differ.

The customer should determine which party is the controller, processor, service provider, or another contractual role for the relevant processing. One organization may play different roles for different activities. Ask the privacy or legal owner to confirm the arrangement and the right contract. The FDE can prepare the data-flow facts and technical settings; the FDE should not decide the legal role or sign terms on the customer's behalf.

Use governance frameworks as working references

AI governance is the set of roles, policies, decisions, records, and review practices an organization uses to direct and oversee AI systems. A framework can help a team ask consistent questions, but it does not automatically make a system safe or legally compliant. Match the framework to the customer's sector, region, risk tolerance, and internal program.

The NIST AI Risk Management Framework (AI RMF) 1.0 is a voluntary framework with four functions: Govern, Map, Measure, and Manage. In plain terms, Govern assigns policies and accountability; Map describes context and risks; Measure evaluates and monitors them; Manage prioritizes responses and improvement. The functions are not a one-time sequence that ends at launch. NIST currently says AI RMF 1.0 is being revised, so check its current status before using it as the basis for a customer program. Its AI RMF page and Playbook provide current entry points.

ISO/IEC 42001:2023 is an international standard for an AI management system. ISO describes it as specifying requirements for establishing, implementing, maintaining, and continually improving such a management system in organizations that develop, provide, or use AI. It is an organizational management-system standard, not a technical test that certifies a particular model's answers. The customer can decide whether to adopt it or seek an assessment. See the official ISO/IEC 42001 page for its scope and current edition.

Regional rules can impose specific duties depending on location, system purpose, risk category, and organizational role. As checked on 5 November 2026, the European Commission's AI Act overview says the Act became applicable on 2 August 2026, with exceptions and phased dates. The Commission lists high-risk provisions for certain sensitive use cases from 2 December 2027 and for AI embedded in regulated products from 2 August 2028 after the 2026 AI Omnibus changes. The Act also has provisions that started earlier. These dates are an orientation point only. The customer must determine whether the law applies, how a system is classified, and what obligations follow, with qualified legal or regulatory support. Recheck the Commission's current page and legislation before relying on a timeline.

The NIST framework and ISO standard are useful ways to structure questions. Neither substitutes for the customer's security controls, data agreements, or local legal advice. When a questionnaire asks whether the system is "compliant," do not answer yes based on using a framework. Describe the controls and evidence that exist, then route the compliance determination to the authorized owner.

Prepare the review packet

The template below helps gather the facts a customer reviewer is likely to need. It is not a standard questionnaire. Fill it with the technical owner, mark unknowns plainly, and attach the customer's approved architecture and data-flow diagrams. Do not include secrets or raw personal data in the packet.

Track each open item with a named owner, requested evidence, due date, and status. A written exception should identify what is not met, why the customer accepts it, which compensating control applies, who approves it, and when it expires or is reviewed. An FDE should not quietly downgrade a required control because a deadline is close. If the decision is unresolved, label it unresolved and explain the deployment impact.

Mistakes beginners make

Answering a security questionnaire from memory. Similar cloud products can have different settings, and customer agreements can override public defaults.

Fix: Confirm the deployed architecture and applicable product terms with the relevant owner. Attach evidence and mark assumptions rather than guessing.

Treating authentication as authorization. A valid user session proves an identity was established; it does not prove that the user can access a particular shipment or invoke every tool.

Fix: Test object-level and function-level permissions on the server for allowed and denied users. Review the test with the API owner.

Assuming a region setting settles every residency question. Backups, logs, support operations, model processing, and subprocessors can involve additional locations or access paths.

Fix: Ask the customer what residency means for this use case, map all data flows, and let the privacy or legal owner interpret the applicable contract and rules.

Calling a framework a compliance certificate. NIST AI RMF is voluntary, and ISO/IEC 42001 describes an organizational management system. Merely referencing either does not prove a product or deployment complies with a law.

Fix: Report the actual controls and records. Ask the authorized security or legal owner to make formal compliance statements.

Recap and try it yourself

Start a customer review with a system map and a data-flow diagram. Mark trust boundaries, data categories, identities, external services, and owners. Enforce authentication and authorization at the API and tool boundary, minimize and protect data, and verify supplier terms before sending customer content. Use DevSecOps practices during delivery and use NIST AI RMF, ISO/IEC 42001, and regional rules as references whose scope and current status need checking. Route legal interpretations and formal approvals to qualified customer owners.

Try it yourself: draw the fictional NorthPort assistant with a dispatcher, customer API, retrieval store, model service, dispatch tool, and log platform. Mark which fields cross each boundary and which party controls the component. Then fill in the review packet with three unanswered questions about data location, tool authorization, and retention. Assign each to a fictional customer role. Write down the evidence that role would need before approving launch, and flag any legal classification question for counsel rather than answering it yourself.

Sources

Checked on 5 November 2026. NIST's AI RMF page, AI RMF Core, and Playbook support the four-function overview, voluntary use, and its current revision status. The official ISO/IEC 42001:2023 page supports its scope as an AI management-system standard. The European Commission's AI Act overview supports the dated, phased timeline; legal applicability and classification remain for qualified review. The Commission's GDPR application guidance supports the controller/processor and contract discussion in the EU context. NIST SSDF SP 800-218 supports integrating security practices into an organization's software development lifecycle. IETF RFC 9700 supports OAuth 2.0 security guidance, and OWASP's API Security Top 10 supports API authorization and abuse review prompts. NorthPort is fictional and uses synthetic data only.