Investigate how to use AI (GitHub Copilot) to answer customer questions in a Request for Information / Request for Proposal, grounded in official Microsoft Learn documentation for Power Platform.
Five RFI requirements were evaluated across all experiments (see requirements.md):
- Automation framework — Logging, exception handling, retry, transaction management
- Availability — 99.9% uptime, 24x7, operational diagnostics
- Audit trail — Change tracking for users, data, configuration, etc.
- SaaS delivery — Browser-based, no local install or plugins
- Migration approach — Structured assessment, conversion, testing, validation, cutover
Each experiment iterated on the prompt to improve source quality, citation accuracy, and output format.
| # | Description | Run in | Key change from previous |
|---|---|---|---|
| 0 | Team baseline — initial answers provided by the team | Manual | — |
| 1 | First AI run using Microsoft Learn MCP Server | VS Code Copilot Chat | Added: search Microsoft Learn for each requirement |
| 2 | Added Power Platform context to queries | VS Code Copilot Chat | Added: "this is about Power Platform" in every query |
| 3 | Restricted to Power Platform sources only | VS Code Copilot Chat | Added: reject Azure-only content and citations |
| 4 | Added citation validation pass | GitHub Copilot CLI | Added: verify each URL resolves and content supports the claim |
| 5 | RFI/RFP paste-ready output with full prompt engineering | GitHub Copilot CLI | Added: professional vendor voice, strict output format, compliance summary, URL verification, multi-query coverage, honest partial-compliance flagging |
| Dimension | Exp 0 | Exp 1 | Exp 2 | Exp 3 | Exp 4 | Exp 5 |
|---|---|---|---|---|---|---|
| Total citations | 5 | 20 | 19 | 9 | 42 | 25 |
| Azure-only citations | 0 | 4 | 5 | 0 | 0 | 0 |
| Power Platform only | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
| URLs verified | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| Content supports claim | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| Validation pass | ❌ | ❌ | ❌ | ❌ | ✅ (visible) | ✅ (silent) |
| Compliance summary | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| RFI paste-ready | Partial | ❌ | ❌ | ❌ | ❌ | ✅ |
| Output tone | Brief vendor | Internal notes | Internal notes | Internal notes | Verbose + tables | Professional vendor |
| Assessment notes | No | Yes | Yes | Yes | Yes | No |
- Source scoping matters. Without explicit "Power Platform only" constraints (Exp 1–2), the model defaults to Azure documentation for uptime/monitoring questions.
- Citation verification is essential. Hallucinated or broken URLs appeared in every experiment before Exp 4 introduced URL fetching and validation.
- Output format must be specified. Without prescriptive format rules, the model produces internal working documents (assessment notes, bullet dumps, validation tables) rather than customer-ready deliverables.
- Multi-query searching improves coverage. Exp 3 found fewer sources by using a single query per requirement; Exp 4–5 used varied phrasings to maximize coverage.
- Honest framing beats overclaiming. Power Platform doesn't have a single "RE Framework" equivalent or automated conversion from competing RPA tools — stating boundaries constructively is more credible than overstating capabilities.
- Iterate, then iterate again. Even Experiment 5 — designed as the "definitive" prompt after analyzing four prior iterations — still had gaps that only surfaced after running it and critically reviewing the output. No single pass, no matter how thorough, catches everything. Each iteration reveals a new class of problems invisible from the previous vantage point.
- Compare iterations and learn from each. The improvements from Exp 0→5 weren't linear. Exp 3 was more honest than Exp 4 about migration gaps. Exp 0's brevity was closer to paste-ready than Exp 4's verbose validation tables. Reviewing all experiments side by side exposed patterns (Azure leakage, broken URLs, wrong migration framing) that no single experiment revealed on its own.
Experiment 5 was designed by analyzing experiments 0–4 and encoding every lesson learned into a single comprehensive prompt. It should have been definitive. It wasn't. Here's what happened and why:
The instruction was reactive, not anticipatory. It fixed every known problem from prior experiments (Azure leakage, broken URLs, wrong tone, missing validation). But it couldn't fix problems that hadn't occurred yet. Three categories of gaps emerged only after running the prompt and reviewing the output:
-
Domain knowledge gaps. The instruction said "RFI paste-ready" but encoded an incomplete mental model of what that means. It specified tone and paragraph format but missed that real RFI responses include a compliance status field (Compliant / Partially Compliant) and a summary table that evaluators scan before reading details. These are RFI conventions that only became obvious when reading the output through a procurement evaluator's eyes.
-
Search strategy gaps. The instruction said "use multiple phrasings" but the actual searches were Power Platform-centric ("Power Platform error handling"). This missed the Flow Exception Rules Framework from the Automation Kit — the closest analog to UiPath's RE Framework — because no search queried for "RE Framework equivalent" or "automation kit framework." For Req5, searching "Power Platform migration" returned results about migrating within Power Platform (between environments), not from a competing platform to Power Platform. The instruction's constraint ("include Power Platform in queries") actually contributed to this blind spot.
-
Writing judgment gaps. The instruction said "every sentence must earn its place" but the output still gave the desktop RPA caveat a full paragraph in Req4 (disproportionate) and framed Req5 as a data migration story when the requirement was clearly about automation platform migration. These are editorial calls that are hard to encode in rules — they require reading the output critically against the original requirement.
The core insight: writing a prompt is a hypothesis about what will produce good output. Running it is the experiment. Reviewing the output is the analysis. You literally cannot see all the failure modes from the instruction side alone — some only become visible in the result. This is why prompt engineering is iterative by nature, not a single-shot optimization.
| File | Description |
|---|---|
| requirements.md | The five RFI requirements |
| instructions.md | Prompt instructions for each experiment |
| experiment0.md | Team baseline responses |
| experiment1.md | First AI run — Microsoft Learn, no scoping |
| experiment2.md | Added Power Platform context to queries |
| experiment3.md | Restricted to Power Platform sources only |
| experiment4.md | Added citation validation pass |
| experiment5.md | Final — RFI/RFP paste-ready with full prompt engineering |