Evidence Agent
Research answers you can inspect
I built a bounded research agent that reads public sources, uses tools, and exposes its observations. A personal Hugging Face Agents Course project by Renchao Wu.
Verified historical run: 9/20 · 45% · local Qwen3-14B · 2026-10-03
Official certificate · Code & run instructions · Evaluation & failures
The live agent uses Qwen3-4B on free ZeroGPU. Its accuracy has not been established by the historical 14B score. This is a course project certificate, not a proctored professional certification.
Ask a focused public research or calculation question. The answer and raw tool trace appear below. Free GPU access may require Hugging Face sign-in and is quota limited. Do not enter secrets or private documents.
These examples execute the real tools now, without an LLM or GPU. They demonstrate budget checking, data checking, and primary-source reading. They are not autonomous agent runs or benchmark results.
Evidence, evaluation, and limits
What was measured
On 2026-10-03, the official course scoring API returned 45.0%: 9 correct out of 20 attempted, under WideChaos. The run used mlx-community/Qwen3-14B-4bit on local Apple Silicon, with the shared research controller. This was one run, not a multi-seed study. The online demo uses Qwen/Qwen3-4B-Instruct-2507; the 14B result is not a measured score for the online model.
- Original official score response
- Official certificate issued to Renchao Wu
- Source revision corresponding to the recorded project
- Fingerprints of the original runtime artifacts
- Official evaluation procedure
The original full model traces and submitted answers are preserved privately. Their fingerprints establish artifact identity, not independent proof of accuracy. The score response is a published copy of the API receipt; the independently hosted certificate confirms course issuance. No complete benchmark answer list or ground truth is shipped with this project. The scorer returned only an aggregate result, so I do not label individual answers correct or incorrect without evidence.
This is a course final-project certificate, not a proctored professional certification. The project was implemented and operated with coding-assistant help under my direction. The result does not establish unaided exam performance or that every course video was watched.
Observed failures
| Observation in original run | Consequence | Current handling / remaining limit |
|---|---|---|
| Five official attachments returned HTTP 404 | Five tasks received an empty answer | Checked again on 2026-10-10: all five still returned 404; probe responses. Missing files cause abstention. This is a dated observation, not a permanent API claim. |
The official course webpage triggered Private network URLs are blocked inside the hosted Space on 2026-10-10 |
Source-reading walkthrough could not read that URL | The private-address block remains enabled; use another public primary source or inspect the explicit error. A URL working locally does not guarantee hosted access. |
A model tool action produced Unterminated string starting at: line 1 column 45 (char 44) |
A tool step was lost | The error and model output are retained; the controller requests a corrected action, within the same finite budget. |
| Video questions produced explanatory refusals instead of exact short answers | Unsupported evidence and inconsistent abstention format | No video decoder or vision model is implemented. Explanatory refusals may still be nonempty answers. |
| A research question ended with an explanation that the source evidence was missing | No supported factual answer | Source retrieval can fail, truncate relevant text, or miss a page. Final answers are model judgments, not validated citations. |
These categories can overlap; they do not explain every incorrect item. The project does not claim that repairing missing files would raise the score by a particular amount.
Non-benchmark demonstration
On 2026-10-10, a new local 14B run with seed 42 selected the calculation tool for 12*35+3*50, observed 570, then returned 570. Full model/tool trace and environment are public. This is a recorded example, not an accuracy study or an online 4B result.
Reproduce a new evaluation
Use the Apple Silicon setup in README, then run:
python local_runner.py --smoke
python local_runner.py --output-dir .local/reproduction --seed 42
This downloads the current public questions, generates actual tool/model traces, and writes a checkpoint and run configuration. Review those files. Submission changes the official leaderboard; it is optional and explicit:
python local_runner.py --output-dir .local/reproduction --submit --username YOUR_HF_USERNAME
The source revision above can be checked out to inspect the historical implementation. Exact regeneration of the old 9/20 result is not guaranteed: the historical run did not record a random seed or complete dependency lock, generation was stochastic, and search results, model repositories, upstream files, and scorer availability can change. New runs record the seed, model ID, Python/package versions, and controller digest. They are new measurements, not replacements for the original receipt. The scorer performs exact-match grading; format errors count against the result.
Capability boundaries
Public HTML and text PDFs; small XLSX tables (up to 1,500 rows and 40 columns per sheet); restricted numeric expressions; English audio via Whisper base.en. Downloads are bounded to 10 MB. The controller permits four tool steps followed by a final answer step; errors consume steps. Raw traces include tool inputs, outputs and model decisions; there is no confidence calibration or citation entailment check.
No image/video understanding, OCR, arbitrary code execution, private repository access, authenticated pages, or employer data. Spreadsheet cached values may be absent or stale; formulas are not recalculated. Search reads only selected excerpts. Web evidence can be malicious or incorrect. Prompt instructions discourage following page instructions and benchmark answer keys, but this is not a hardened prompt-injection sandbox. Public-URL checks block obvious private destinations and recheck redirects; DNS rebinding is a remaining limitation. This educational demo is not suitable for untrusted multi-tenant production deployment or consequential decisions.
Free ZeroGPU can require sign-in and exhaust its quota. CPU tool walkthroughs remain available without model inference; they are explicitly labeled and do not contribute to the benchmark score. The public evaluation cache is shared and ephemeral; do not put private data in any field.
Run the current 4B agent on official questions, inspect traces, then submit separately. Submission requires the Space owner. Checkpoints are shared across visitors and disappear on restart; this is an educational evaluator. No benchmark answers are bundled. Audio uses English Whisper on CPU; images/video and Python execution are unsupported.