Step 1: Modeling the Cost of Showing a Suggestion#

The starting point is simple: showing a suggestion is not free. Even before a user decides whether to accept it, the system has already paid a generation latency cost and the user has already paid a verification cost.

If the suggestion is shown, the expected total time can be written as:

Here:

• $\tau$ is the time spent generating the suggestion
• $c_{\text{verify}}$ is the time spent reading and checking it
• $c_{\text{edit}}$ is the time needed if the user accepts but still edits
• $c_{\text{write}}$ is the time needed if the user rejects and writes manually
• $p$ is the probability that the suggestion is accepted

This equation matters because it shifts the problem away from raw model confidence and toward expected user effort.

Step 2: Turning Time into Utility#

The next move is to compare this with the baseline of not showing anything at all. If the no-suggestion path costs $c_{\text{noshow}}$, then the utility of showing becomes:

Substituting the previous expression gives:

This is the key simplification: once the time costs are fixed, the whole decision collapses onto the acceptance probability $p$.

Step 3: Deriving the Threshold for Intervention#

If writing from scratch is the default fallback, then $c_{\text{noshow}} = c_{\text{write}}$. Under that assumption, showing is beneficial only when:

which gives the threshold:

That threshold is what I found most elegant in the model. It says the system does not need a vague sense that a suggestion is “probably useful.” It needs something sharper: the predicted acceptance rate must be high enough to overcome the combined burden of generation and inspection.

In other words, a coding assistant should intervene only when the expected savings justify the interruption.

Step 4: Why a Scalar Probability Is Enough#

Another useful consequence of the model is that the raw context itself is not needed once it has been compressed into an estimate of acceptance.

Suppose the user state depends on context features $x_t$ and a candidate suggestion $s_t$. The intervention decision can still be made entirely from the scalar prediction $p_t^*$, because all of the contextual complexity has already been folded into that number.

That makes the control problem cleaner. The system does not need to reason over the full state every time it decides whether to show. It needs a reliable estimator of acceptance.

Step 5: Why Learning Is Necessary#

Of course, the real acceptance probability is not directly observable. It depends on latent user state, which means the system has to learn an approximation such as:

or, more explicitly,

This is where the modeling in the PDFs becomes especially important. The decision rule is not useful by itself unless the estimated probability is reasonably calibrated. If $\hat{p}$ is systematically too optimistic, the system will over-intervene. If it is too conservative, useful suggestions will be suppressed.

So the theoretical threshold only becomes operational once it is paired with a predictive model that can estimate acceptance from observable information.

Step 6: Uncertainty and the Two-Stage Policy#

One of the nicest extensions in the modeling section is the move from a single-stage decision rule to a two-stage one.

If a lightweight context-only model already predicts with high certainty that a suggestion will be accepted or rejected, then it may be wasteful to generate a full suggestion just to confirm what is already obvious. But if the prediction is highly uncertain, then generation becomes worth paying for because it can change the decision.

That intuition can be described using entropy: when $\hat{p}$ is near $0.5$, uncertainty is high; when it is near $0$ or $1$, uncertainty is low.

So the policy becomes:

• use a cheap context-only estimate when confidence is strong
• escalate to full suggestion generation only when uncertainty is high

This makes the system more computationally disciplined. It does not spend expensive generation effort on every interaction, only on ambiguous ones.

Step 7: Why Acceptance Rate Is a Bad Proxy#

The final modeling insight I wanted to preserve from the PDFs is the critique of optimizing for acceptance alone.

If suggestion length is denoted by $\ell$, an acceptance-oriented objective can be written as:

This type of objective can favor suggestions that are easy to accept because they are short and low-risk. But short, low-risk completions are not necessarily the ones that save the most time.

If the actual goal is time saved, the utility of an accepted suggestion is modeled as:

so the expected time saved becomes:

Since $\alpha > 0$ does not change the maximizer, the effective objective is:

If $\ell_{\text{acc}}$ maximizes $R(\ell)$, then the condition for the time-optimal choice to move to the right is:

Equivalently,

or in ratio form,

That distinction captures a common failure mode in coding assistants. A system that optimizes for acceptance alone may learn to surface tiny, safe, nearly trivial completions. Those completions boost the metric, but they do not meaningfully improve productivity. The metric looks healthy while the user experience remains shallow.

For me, this was the conceptual center of the project: a good coding agent should optimize real utility, not just superficial agreement.

Step 8: Why One Reference Solution Is Not Enough#

Once the suggestion policy is defined, the next question is where those suggestions come from. A natural baseline is supervised fine-tuning on reference solutions. If a model assigns probability $\pi_\theta(y \mid P)$ to a solution $y$ for problem $P$, then the standard SFT objective is:

In a finite dataset, that becomes:

For a sequence $y = (y_1, \ldots, y_T)$, the model factorizes autoregressively:

Therefore, the token-level negative log-likelihood expands as:

This is elegant, but it has a limitation that matters a lot for code. Many programming problems admit multiple correct implementations. If training pushes hard on only one reference trajectory, it can reduce probability mass on other valid solutions. In other words, SFT is faithful to the dataset, but not necessarily to the full solution set.

That is one of the reasons I found reward-based training more compelling for this project. Passing tests cares about semantic correctness, not stylistic imitation of a single reference answer.

Step 9: Binary Reward vs. Fractional Reward#

Once code is executable, reward design becomes the next modeling choice. If a candidate passes $k$ tests out of $n$, two obvious options are:

The fractional reward contains partial credit. It tells the model whether it is moving closer to a fully correct program even when it still fails some tests. That can make training denser and more informative.

At the same time, binary reward has a valuable strictness. It discourages opportunistic behavior that merely targets subsets of tests and instead forces the system to care about complete correctness. For code generation, that hard threshold often aligns better with how solutions are actually judged.

Step 10: What a Completion Pipeline Really Needs#

Before any reward can be assigned, the system needs a concrete procedure for producing a candidate program. In practice, that means more than sampling tokens.

The generation pipeline has to:

1. render the problem into the right prompt template
2. tokenize and initialize the model state
3. sample autoregressively under a decoding rule
4. stop on conditions stronger than just max tokens
5. decode the result into raw text

The stop conditions matter more than they first appear. End-of-sequence tokens, fenced code block termination, or explicit end markers all change the quality and cleanliness of outputs. If these controls are sloppy, evaluation becomes noisy before training even begins.

Step 11: Why Prompt Rendering and Extraction Are Part of the Environment#

I ended up treating prompt rendering and code extraction as first-class parts of the system, not incidental preprocessing.

If a model is paired with the wrong chat template, several things can go wrong:

• role tokens may be interpreted as plain text
• system instructions may lose their priority
• the model may produce malformed or out-of-distribution outputs

Similarly, code extraction from raw model text has to be deterministic. If a response contains explanations and multiple code fences, the evaluator needs a rule for choosing exactly one program string or returning failure. This is not just cleanup. It directly affects reward.

That is why formatting became part of the environment design. A logically correct solution that is wrapped in unusable structure can still fail extraction and receive no useful reward.

Step 12: Why Safe Evaluation Requires a Sandbox#

The moment code is executed in the loop, safety becomes non-negotiable. Running generated code through exec() on the host machine is not acceptable in any serious training pipeline.

There are at least two classes of failure:

• accidental failures, such as infinite loops, uncontrolled writes, or resource exhaustion
• adversarial failures, such as exfiltration of secrets, destructive shell commands, or malicious network behavior

This is why the sandbox is not auxiliary infrastructure. It is part of the learning environment itself. Without safe execution, automated code reward is not robust enough to trust.

Step 13: Reward Shaping and Its Tradeoffs#

The shaped reward I used conceptually is:

and in the training setup this becomes:

with a small formatting coefficient.

I found the tradeoff here interesting. A negative formatting term helps the model distinguish malformed output from merely incorrect logic, which can accelerate learning of basic output discipline. But too much weight on format can also create a distorted incentive, where the model learns to produce safe-looking structure without solving the actual task well.

That tension is exactly the kind of thing I wanted this project to make visible: reward design is never neutral. It shapes what the model learns to care about.

Step 14: From Reward to Policy Gradient#

Once a sampled completion $y = (y_1, \ldots, y_T)$ receives a scalar reward $R(y)$, the objective is to maximize expected reward:

Writing the expectation explicitly:

Using the log-derivative trick, $\nabla \pi = \pi \nabla \log \pi$:

Switching back to expectation notation:

Since sequence probability factorizes over tokens,

so the final policy gradient becomes:

This is the bridge between automatic evaluation and learning. Once reward exists, token probabilities can be updated in the direction of solutions that perform better under execution.

Step 15: Baselines, Advantages, and Variance Reduction#

Raw rewards are noisy, so the next refinement is to subtract a baseline $b(P)$ and define:

The key reason this works is:

So subtracting the baseline leaves the expected gradient unchanged, while reducing variance. Intuitively, the model is no longer being told only “how good was this sample?” but rather “how good was this sample relative to what was expected for this problem?”

For grouped sampling, this becomes even cleaner. If we sample $G$ completions for the same prompt and define

then

because

This means the update is purely group-relative. The model is pushed toward completions that outperform their peers under the same prompt, and degenerate groups where all rewards are equal provide no learning signal at all.

That logic flows directly into GRPO. It is a neat fit for code generation because code rewards are sparse, high-variance, and naturally comparable within a prompt-conditioned group.

Running SimpleCoder#

The first implementation is packaged as a command-line tool. From the project root, the workflow is straightforward:

1
pip install -r requirements.txt
2
pip install -e .
3
simplecoder "create a hello.py file"
4
simplecoder --use-rag "what does the Agent class do?"
5
simplecoder --use-planning "create a web server with routes for home and about"
6
simplecoder --interactive

This deployment model keeps the agent lightweight. It behaves more like a controlled coding utility than a general-purpose assistant shell.

Running the RL Pipeline#

The second implementation depends on sandboxed execution, so its setup is slightly more infrastructural. After installing requirements, I start the Sandbox Fusion server and then run train.py.

1
python -m pip install -r requirements.txt
2
docker run -it -p 8080:8080 -v ./sandbox_config/local.yaml:/root/sandbox/sandbox/configs/local.yaml volcengine/sandbox-fusion:server-20250609
3
python train.py

The same training script supports two backends:

• backend=tinker for remote training
• backend=local_gpu for local LoRA fine-tuning

That split made the project easier to iterate on, since the same overall training logic could be tested under different compute conditions.

Part I: SimpleCoder#

Part II: Code RL Post-Training#