Pattern

Aiming to repeat and improve on a pattern here that i’ve found effective, that being email triage and routing.

The system and idea here is that you:

• Intercept emails landing in a busy inbox
• Feed the emails into an LLM
• Provide the LLM tools to search for more context relevant to that email
• Have the LLM make a decision on how to treat the email
• Feed that decision back into the right tools or system that can act on that decision (i.e. forward to person or group X).

Pattern Implementation

Using the system I built as an example (M365 focused), the steps components were:

1. Intercept the emails. Power Automate cloud flow trigger on shared inbox.
2. Extract the content. Power Automate steps to extract email sender, title and content
3. Extract the attachments. Power Automate step to extract text from attachments via M365 AI builder OCR
4. Feed all email content to an LLM. Power Automate step to send content via HTTP to the LLM
5. The LLM that receives and processes the content. Azure AI Foundry Prompt Flow Deployment
6. Give the LLM the tool to search for more context. Prompt flow has a tool call to a safe read-only API bridge that can access specific read-only endpoints of a REST API enabled database
7. LLM makes a decision on the routing target for that email. Prompt flow uses the email content and context, as well as the tool call outputs to make a decisions
8. The LLM sends the decision back to the system that can act on that information. LLM sends decision back to another Power Automate flow with a HTTP trigger
9. Parse decision and implement guards. That Power Automate flow parses and gates the decision to ensure the response is within bounds (what’s expected) and provides guards to ensure it’s only sent internally (i.e. no outward bound automated emails to customers)
10. The system forwards the email. Power Automate step to send the email to the desired inbox

As an extension to this I found I needed a way of capturing and monitoring the output of this decision, to enable a feedback loop for improvements. With that the other components are:

1. Save the context (email and tools calls), LLM decision making, and output of each run. Power Automate step to capture all decisions and outputs at the start of the 2nd Power Automate flow that parses, gates and forwards emails. These outputs and decisions are sent to a SharePoint hosted excel spreadsheet
2. A feedback loop that identifies and flags incorrectly forwarded emails. A button in the forwarded email that lets users report when the LLM made a wrong decision. The wrong-decision would be marked/saved against the excel spreadsheet list of decisions

Generalised Pattern Components

For this to work for any system or implementation, the core components needed are:

• Workflow Automation: Something that can interact with the delivery or task pipeline sufficiently to complete the task (e.g. can intercept, read and forward emails).
• The Agent: An LLM that can receive and act on requests
• Tools and Context Access: A secure and fail-safe way for that LLM to interact with relevant context and systems via tools
• Guardrails: that restrict and control how the output of the LLM interacts back with the workflow automation
• Measurement and Feedback: to enable measurement of effectiveness for iteration and improvement

Next posts will be a live demonstration of this model.

This site’s purpose is to explore and experiment with using agents in a useful way. That is, agents that are solving real problems or providing value in a measurable way, beyond the cost of managing those agents.

Going to roughly categorise the usage of these agents into two categories:

1. Agents that are building novel things; such as software, content or other creative endeavours.
2. Agents that have been put to use solving a measured repeatable task, integrated into businesses and systems, and saving measurable hours and resources for people.

That’s about it for now. Next posts i’ll be focus on implementing something in that second category and see what falls out.