When I first added hidden test generation to ForkSpace — my real-time collaborative DSA practice platform — the obvious approach seemed straightforward: give the LLM the problem statement, ask it to generate some edge case inputs, and ask it to also tell me the expected output for each one. Then run the user's code, compare outputs, done.

It failed. Badly.

For a simple array reversal problem, it worked fine. But the moment I tested it on a graph problem with specific connectivity constraints, the LLM confidently generated an input — and just as confidently invented a wrong expected output. My system was now telling users their correct solutions were wrong. That's not a testing assistant. That's a gaslighting machine.

I had to tear the whole thing down and rethink it from scratch.

Why the obvious approach breaks

Here's the thing about LLMs and competitive programming: they're excellent at understanding problem structure. They can look at a problem statement and immediately reason about what kinds of inputs would stress a solution — boundary values, empty cases, overflow triggers, worst-case graphs. That's genuine reasoning ability.

But expected outputs? For non-trivial problems, LLMs hallucinate. Not always. Not obviously. Just enough to be dangerous.

A wrong expected output in a test suite is worse than no test at all. At least with no test, you're not actively misleading the user. But when your "automated judge" says their O(n log n) solution failed on a specific graph input — and the expected output your LLM invented was simply wrong — you've just destroyed trust in your entire system.

The root cause is this: LLMs are planners, not compilers. Asking an LLM to produce a correct expected output for a complex input is asking it to mentally simulate code execution. Sometimes it can. Often it can't. And it has no reliable way to tell you which situation you're in.

So the architecture needed to reflect that reality.

The split that fixed everything

The system I ended up building has one core principle:

The LLM plans. Code generates. Piston executes. Nobody asks the LLM to be a judge.

Here's what that looks like in practice.

Step 1: LLM does only one job — planning

The LLM receives the problem statement, constraints, and sample I/O. It returns a JSON array of test plans. Not test inputs. Plans.

Here's the actual prompt:

You are a test planning assistant for competitive programming.
Return ONLY valid JSON array. No markdown. No prose.
Generate 4 concise hidden test plans.
Do NOT provide expected outputs.
Do NOT provide raw stdin strings.

Schema:
[
  {
    "category": "boundary|stress|robustness|randomized",
    "description": "short description",
    "bugClass": "off-by-one|overflow|empty case|wrong loop bounds|unknown",
    "inputSpec": { "type": "array|string|graph", ... }
  }
]

Problem statement:
${problemStatement}

Constraints:
${constraints}

Sample I/O:
${sampleIO}

Two lines in that prompt are doing most of the architectural work:

Do NOT provide expected outputs. This is the entire philosophical stance of the system in one instruction. You're explicitly telling the LLM what it's unreliable at and removing it from that responsibility entirely.

Do NOT provide raw stdin strings. This keeps the LLM in the abstract layer. It describes what to generate — the shape, the category, the intent. Your code decides how to format it. Same plan always produces consistent, valid input. LLM-generated stdin has unpredictable whitespace, newline handling, and formatting variance that breaks execution in ways that are silent and maddening to debug.

The simplest mental model: the LLM is a senior engineer writing a test plan doc. It thinks about categories and edge cases. It never touches a compiler.

Step 2: Deterministic code handles generation

The plans come out of the LLM looking like this:

{
  "category": "boundary",
  "description": "single element array",
  "bugClass": "off-by-one",
  "inputSpec": { "type": "array", "size": 1, "valueRange": [1, 1000] }
}

inputGenerator.js takes that inputSpec and constructs the actual stdin string. Right now it handles arrays, strings, and graphs. The point is: generation is fully deterministic and constrained. No LLM variance, no formatting surprises. The same plan produces the same valid input every time.

This is the "LLM for planning, code for generation" pattern. It's much safer than "LLM generates everything" because it plays to each component's actual strengths.

Step 3: Piston runs the code, classifies the result

The user's code plus the generated stdin goes to Piston for execution. Piston is an isolated code execution engine — it runs C++, Python, and JavaScript cleanly.

After execution, classifyOutput labels each result:

• pass / fail — only for verified tests where a trusted expected output exists

• timeout — execution exceeded the time limit

• crash — runtime error, segfault, bad memory access

• stress-only — no expected output exists, so we can't pass/fail, but the execution behavior is still a signal

That last category is the key design decision.

Verified tests vs stress tests — why the split matters

The system runs two kinds of tests and it's important to understand why they're separate.

Verified tests are built from existing room sample cases plus any curated additions. These have trusted expected outputs. Pass/fail is strict and reliable.

Stress tests come from the AI-generated plans. They have no ground truth output. We never invented one. Instead, we run the user's code and watch what happens — does it crash? Does it timeout? Does it produce suspicious output? These are signals, not verdicts.

This is intellectually honest in a way most automated testing tools aren't. The system won't tell you your solution is wrong unless it actually knows the right answer. For stress tests, it tells you: "your code crashed on this input" or "your code timed out on this graph" — which is genuinely useful information without requiring a ground truth.

There's also a promotion flow: if a stress test produces interesting behavior, the user can adopt the generated output as a new sample case for debugging in the room. The "correct" answer emerges from human judgment, not LLM invention.

The part I didn't expect to matter: bugClass

When I designed the prompt schema, I added bugClass as a metadata field mostly for display purposes. It turned out to be the most valuable field in the whole system.

Here's why: hidden test outcomes feed directly into session intelligence reports. After a practice session, the system summarizes what happened — strengths, gaps, patterns. And bugClass is what makes those patterns specific.

Instead of "you failed 3 hidden tests," the report can say: "you failed 3 tests classified as off-by-one — check your loop termination conditions." That's the difference between a score and a learning signal.

The LLM isn't just planning test shapes. It's labelling the intent of each test — what class of bug it's designed to expose. That label flows downstream through execution and into the user's learning outcome. I didn't plan that pipeline upfront. It emerged from having the right field in the schema from the start.

What the full flow looks like

Workspace UI
  → /api/hidden-tests/generate
      → room problem statement + constraints + samples
      → LLM planner (Gemini, fallback to safe defaults)
      → structured input generator
      → save test plans

Workspace UI
  → /api/hidden-tests/run
      → fetch saved tests
      → Piston execution
      → classify output (pass/fail/timeout/crash/stress-only)
      → save results

Hidden test results
  → session intelligence report
      → bugClass pattern aggregation
      → learning signal surface

One more reliability detail worth mentioning: if MongoDB is unavailable, the router falls back to an in-memory room map. If Gemini is unavailable or hits quota, the planner falls back to safe default plans. The feature degrades gracefully instead of breaking. In a tool people use during timed practice sessions, silent failures are unacceptable.

What this system is not

I want to be honest about the limitations because I think intellectual honesty is what separates useful technical writing from marketing.

This is not a full competitive programming judge with official expected answers for all hidden tests. It won't give you the same pass/fail guarantee as Codeforces or LeetCode's hidden judge. It can't — it doesn't have a reference solution, and it's not going to pretend an LLM-invented answer is ground truth.

What it's actually good at: surfacing crashes, timeouts, edge case behavior, and suspicious output patterns. For collaborative practice and mock interviews — which is what ForkSpace is built for — that's exactly the right scope.

The input generator currently handles arrays, strings, and graphs. More structured types would meaningfully expand coverage. That's the most obvious next build.

What I'd build next

The bugClass field opens up a feedback loop I haven't closed yet.

Right now, the LLM plans tests based on the problem statement. It doesn't know anything about what kinds of bugs users on this platform actually hit most frequently. But that data exists in the session reports.

The natural next step: feed aggregated failure patterns back into the planning prompt. If users consistently hit overflow bugs on problems with large integer constraints, weight the planner toward generating more overflow-class tests for similar problems. The system gets smarter about what to test based on what people actually get wrong.

That's a loop from session data → planner prompt → test quality → session data. I haven't built it yet. But naming it felt important because it's where this kind of system eventually has to go to be genuinely useful.

The one-line version

If I had to reduce the entire architecture to a single principle:

Ask the LLM what to test. Ask your code to build it. Ask a real executor to run it. Never ask the LLM to grade it.

That split — planning vs generation vs execution vs judgment — is what makes the system reliable. Each component does what it's actually good at.

If you want to poke around the implementation, the main files are testPlanner.js, inputGenerator.js, pistonRunner.js, and hiddenTestRoutes.js in the ForkSpace repo.

If you've built something similar or hit different failure modes with LLM-generated tests, I'd genuinely like to know what you found. The failure modes are where the interesting design decisions live.

ForkSpace is a real-time collaborative DSA practice platform for peer sessions, mock interviews, and mentoring. If you're preparing for placements and want to practice with a partner, give it a try.