| title | Decision math, explained simply | |||||||
|---|---|---|---|---|---|---|---|---|
| description | A plain-language walkthrough of how agent-learning turns explicit choices into probabilities and uses measurable outcomes to improve the next decision. | |||||||
| author | Microsoft | |||||||
| ms.date | 2026-08-09 | |||||||
| ms.topic | overview | |||||||
| keywords |
|
|||||||
| estimated_reading_time | 20 |
This is the friendly, no-scary-symbols version of math.md. Every idea here has a matching grown-up formula in that file, so once an idea "clicks," you can peek at the real math and recognize it.
The big picture: this SDK keeps a small decision policy beside an existing agent. The policy chooses among explicit actions, watches how execution turns out, and gradually changes the next choice. Almost all the math is about two things:
- Making a choice when you're not 100% sure (turning "scores" into "chances").
- Learning from what happened (nudging yourself to do more of what worked).
Keep that in mind and everything below is just details.
Grown-up name: softmax. See math.md.
Imagine you have three snacks and you give each a score for how much you like it:
- Pizza: 2
- Apple: 1
- Broccoli: 0
You don't want to always pick pizza (boring, and maybe you'd miss out). You want pizza to be most likely but still give the others a chance. Softmax turns those scores into chances that add up to 100%:
- Pizza → about 67%
- Apple → about 24%
- Broccoli → about 9%
The trick: bigger scores get much bigger chances, but nothing ever drops to a flat zero.
One-sentence version: Softmax is a fair spinner wheel where the choices you like get bigger slices.
Tiny grown-up peek: you raise a special number
$e$ (about 2.718) to the power of each score, then divide each by the total. Higher score → way bigger slice.
Grown-up name: contextual softmax / W · phi. See math.md.
Sometimes the best choice changes depending on what's going on. You'd pick a different snack at a birthday party than before bedtime.
The "situation" gets written down as a list of numbers called phi (say it "fye"). For example: is it morning? is it hot out? am I hungry? → [1, 0, 1].
The policy keeps a little grade book (called W) that says how much each part of the situation should push each choice up or down. It multiplies the situation by the grade book to get fresh scores, then uses the same softmax spinner from Step 1.
One-sentence version: Same spinner wheel as before, but the slice sizes change based on what's happening right now.
Grown-up name: inverse-CDF sampling. See math.md.
Once you have chances (67% / 24% / 9%), how do you actually pick? Imagine a raffle:
- Pizza gets tickets numbered 0–66.
- Apple gets 67–90.
- Broccoli gets 91–99.
You spin a random number from 0 to 99 and see whose tickets it landed in. More tickets = more likely to win, but the little guys can still surprise you.
One-sentence version: Give everyone raffle tickets based on their chances, then draw one ticket.
Grown-up name: REINFORCE-with-baseline. See math.md.
This is the heart of the whole thing. After the agent executes the policy's choice, the observed result gets a reward, like points in a game. Good result = high points. Bad result = low points.
The rule is simple: do more of what earns points, less of what loses them.
But there's a clever twist called the baseline. The baseline is "what I usually score." You only get excited if you beat your usual. Two examples:
- You normally score 5. This time you got 8. That's better than usual (+3), so do more of that choice.
- You normally score 5. This time you got 2. That's worse than usual (−3), so do less of that choice.
Without a baseline, everything looks like a win and the policy cannot tell great from just-okay. The baseline is the "compared to normal" ruler.
One-sentence version: If a choice does better than your usual, lean into it; if it does worse, back off.
Tiny grown-up peek: the nudge is
learning-rate × (reward − baseline) × (how surprising the choice was). The "learning rate" is just how big each step is.
Grown-up name: entropy and the entropy bonus. See math.md.
If the policy always picks pizza, it will never discover that tacos work better. So we add a small reward just for keeping an open mind and trying different things now and then.
Entropy is a fancy word for "how spread out are my choices?"
- Always picking one thing → low entropy (closed-minded).
- Giving everything a fair shot → high entropy (curious).
The "entropy bonus" gently pushes the policy to stay a little curious so it does not get stuck too soon.
One-sentence version: A little reward for staying curious, so you don't lock onto one answer before exploring.
Grown-up name: EMA (exponential moving average) baseline. See math.md.
Remember the "usual score" (baseline) from Step 4? How do we keep it updated? We use a moving average — an average that leans toward recent results.
Think of your grade in a class: it's mostly your old average, but each new test nudges it a bit. One great test doesn't erase the whole year, but it moves the number a little.
One-sentence version: Keep a running average of your scores, letting recent results count a little more.
Grown-up name: importance weighting. See math.md.
Sometimes the policy learns from old memories, choices captured by an older policy version. That needs care because the active policy might not make those same choices today.
So it puts old memories on a scale: memories that still match how it thinks today count fully; memories that don't match get counted less. There's also a cap so no single old memory can shout too loudly.
One-sentence version: When learning from the past, trust memories more if they still match how you'd act today.
Grown-up name: reward shaping. See math.md.
The decision outcome gets graded by several scorers (Did it resolve the intent? Did it follow the task? Did it finish the job?). Each scorer returns a grade from 0 to 1. We need to squish all those into one final score.
Step A — turn grades into good/bad points. A grade of 0.5 is "meh" (zero points). Above 0.5 is good (plus points); below is bad (minus points):
- Grade 0.8 → +0.6 points (nice!)
- Grade 0.5 → 0 points (meh)
- Grade 0.3 → −0.4 points (not great)
Step B — add them up, but some scores matter more. Each scorer has a "weight" (importance). Multiply and add.
Step C — add bonuses and penalties. Was it too slow? Small penalty. Did it route the request correctly? Small bonus.
Step D — cap it. The final score is squeezed to stay between −1 and +1 so it's never wild.
One-sentence version: Turn each grade into good/bad points, add them up (some count more), then cap the total.
Grown-up name: sigmoid / logistic function. See math.md.
Some parts of the SDK answer yes/no questions, like "Did this result actually help?" They start with a plain number that could be anything, and they need to turn it into a confidence from 0 to 1 (0% sure to 100% sure).
The sigmoid is a machine that squishes any number into that 0-to-1 range using a smooth S-shaped curve:
- A big positive number → close to 1 ("almost certainly yes").
- 0 → exactly 0.5 ("total coin flip").
- A big negative number → close to 0 ("almost certainly no").
One-sentence version: A squisher that turns any number into a "how sure am I?" between 0 and 1.
Grown-up name: dot product. See math.md.
How does the SDK turn a situation into a single number to feed the squisher? With a weighted vote.
Say the clues are [2, 5] and their importances are [3, 1]. You multiply each pair and add:
Clues that matter more (bigger importance) push the final number more.
One-sentence version: Multiply each clue by how much it matters, then add it all up.
Grown-up name: gradient descent. See math.md.
How do the yes/no parts learn the right importances? By practicing on examples and fixing mistakes a little at a time.
Imagine tasting soup and adjusting the salt: too bland → add a pinch; too salty → add water. You never fix it in one giant dump; you nudge and taste, nudge and taste. Each round the guesses get a little closer to right.
The learning rate is how big each pinch is. Too big and you overshoot; too small and it takes forever.
One-sentence version: Practice, check how wrong you were, and nudge the settings a tiny bit in the better direction — over and over.
Grown-up name: multinomial logistic regression / router. See math.md.
Sometimes you must pick one option out of many — like a receptionist deciding which department should handle your call.
The router looks at the situation, gives every option a score, runs the softmax spinner (Step 1) to get chances, and picks the top one. If no option looks good enough (the best chance is still low), it's allowed to say "I'm not sure" instead of guessing — that's called refusing, and it's a feature, not a bug.
One-sentence version: A smart receptionist that scores every option, picks the best — and can say "not sure" if nothing fits.
Grown-up name: cosine similarity. See math.md.
The router has a second way to choose: comparing arrows. Picture each option and your situation as arrows.
- Arrows pointing the same way → very similar → score near 1.
- Arrows at a right angle → unrelated → score near 0.
- Arrows pointing opposite → opposites → score near −1.
Cool part: it only cares about the direction the arrow points, not how long it is. So "I love dogs a lot" and "I love dogs a little" point the same way — both are about loving dogs.
One-sentence version: Two things are similar if their arrows point the same direction, no matter how long the arrows are.
Grown-up name: hashing-trick bag-of-words. See math.md.
Computers can't read words, only numbers. So we count words — but with a twist.
A normal way is to keep a giant dictionary listing every word. Instead, we use a magic sorting hat: each word gets tossed into one of, say, 1,000 numbered buckets. Then we just count how many words landed in each bucket. That list of bucket-counts becomes the numbers the classifier reads.
Why the sorting hat? Because you never have to build or store a dictionary — the hat always sends the same word to the same bucket.
One-sentence version: Toss every word into a numbered bucket and count the buckets — no dictionary needed.
Grown-up name: TF-IDF. See math.md.
Not all words are useful. Words like "the," "a," and "is" show up everywhere and tell you almost nothing. Rare words that pop up a lot in one message are the juicy ones.
TF-IDF gives each word an importance score using two ideas:
- Shows up a lot in this message? → more important.
- Shows up in basically every message? → less important (probably boring, like "the").
One-sentence version: A word matters if it's common in this message but rare everywhere else.
Grown-up name: normalization. See math.md.
Different scorers produce values on different scales. One gives stars from 1 to 5; another gives 0 or 1. To compare fairly, we stretch every value onto the same 0-to-1 ruler.
For the 1-to-5 score, we slide it down and shrink it:
- 5 stars → 1.0 (perfect)
- 3 stars → 0.5 (middle)
- 1 star → 0.0 (worst)
Now every grade speaks the same language before we mix them in Step 8.
One-sentence version: Stretch every grade onto the same 0-to-1 ruler so comparisons are fair.
Here is the whole decision loop in one breath:
- A situation comes in, written as numbers (phi, Step 2).
- The policy scores each choice and turns scores into chances (softmax, Step 1).
- It draws from a weighted raffle to pick (sampling, Step 3).
- The result gets evaluated by scorers (sigmoid yes/no answers, Step 9), with each score stretched onto the same ruler (normalization, Step 16).
- The grades are mixed into one reward (reward shaping, Step 8).
- The policy compares that reward to its usual (baseline, Steps 4 & 6) and nudges the next decision toward what worked (learning, Steps 4 & 11), while staying a little curious (entropy, Step 5).
- Over many rounds, it gets steadily better.
That's it. Everything in math.md is just a precise way of writing down these seven simple ideas.
"The agent has a spinner for a recurring decision, and better options get bigger slices. After the selected action runs, the outcome earns points. If that choice beat its usual score, its slice gets bigger next time; if it did worse, the slice shrinks. The policy keeps some room to explore, and the foundation model stays unchanged."