I tried running someone else’s model without giving it the answers.

This is a local prototype using synthetic data. It is not a production security claim.

The radiotherapy challenge I ran before accepted DICOM result files. My evaluator owned the code, so the problem was fairly contained: check the file, calculate the score, put it on the leaderboard.

A general medical AI challenge is different. People may submit a Docker image containing arbitrary code. I wanted to see how little access that image could be given while still producing a prediction.

What I made

The container can read a test-input directory and write to one output directory. It cannot see the hidden truth. Scoring happens later, outside the container. Network access is disabled; the root filesystem is read-only; Linux capabilities are dropped; and the process runs as an unprivileged user with limits on memory, CPU, PIDs, time, and output size.

submitted image → read-only input → predictions.csv
hidden truth → host-side scorer → score + audit.json

The audit record keeps the image ID, file hashes, limits, runtime, exit code, and bounded logs. That tells me what was evaluated. It does not prove that Docker is a perfect sandbox.

What it still does not solve

A container-runtime or kernel bug can cross the boundary. Giving a job access to a GPU creates another attack surface. A carefully designed output can leak information about the test set. Resource limits can also fail in less obvious ways than simply using too much RAM.

Then Docker broke

The scoring tests passed. The isolation command looked right. The first image build failed inside Docker Desktop’s BuildKit storage with an I/O error.

I could have deleted caches until the demo turned green, but that would have hidden the most useful part. A laptop development environment and a dependable evaluation worker are not the same thing. The failure belongs in the test record.

My next run will be deliberately unpleasant: attempts to reach the network, write outside the output directory, fork too many processes, run past the timeout, and return malformed or oversized predictions. After that I can connect the runner to the existing Orcaton queue.

Why I care

Medical datasets often cannot be handed to every participant. In that situation the model has to travel to the data. A competition site stops being a download page and starts becoming an evaluation operator. That seems like a more useful thing for me to build.