AI Engineering · Backend & cloud
LiveDatafact
A serverless engine that turns a Google Form and a chosen persona into coherent, human-plausible survey responses at scale.
- Role
- AI & backend engineer
- Timeline
- 2025 — 2026
- Status
- Live at datafact.site.
- Core stack
- AWS Lambda · AWS Step Functions · API Gateway (HTTP)
What it is
Datafact takes a Google Form URL and a chosen persona, and produces survey responses that read as though they came from that kind of person — consistent from one question to the next, and submitted straight into the form.
The problem
Testing a survey instrument, a form pipeline or a dashboard needs response data before any real respondent exists. Hand-written rows are uniform and obviously synthetic; they never exercise the messy middle of a distribution, which is exactly where the pipeline breaks.
How it went
01A pipeline you cannot test is a pipeline you do not trust
The trigger was mundane. You build a form, a pipeline behind it and a dashboard on top, and then you cannot tell whether any of it works, because there is no data — and there will be no data until you ship it to real people, which is exactly the moment you would like to already know.
The obvious workaround is to type a few rows in by hand. Those rows are useless. They are uniform, they agree with each other, and they never produce the awkward middle of a distribution where the pipeline actually breaks.
02Making the data plausible was the whole problem
Generating a response is trivial. Generating two hundred that behave like two hundred different people, each internally consistent from the first question to the last, is not. A respondent who is careful about one answer and careless about the next is not a person; it is an artefact, and anything downstream looking for structure will find none.
So coherence is enforced per respondent, up front, rather than reconciled at the end. Each one is an independent unit of work, which is also what let the whole thing fan out concurrently without a single bad generation poisoning the batch.
Responses that all arrive in the same second look like exactly what they are.
03Serverless was a trade, not a shortcut
The load is spiky by nature: nothing for hours, then a burst. Lambda behind Step Functions absorbs that and costs nothing in between, which is the right shape for a system that sits idle most of the day.
What it costs is that a run stops being one request and one response. It becomes a state machine you have to watch. I would make the same trade again — but I would design how to watch a run before writing the run.
Architecture
The decisions
A fully serverless, event-driven backend
Load is spiky by nature — nothing for hours, then a burst of concurrent generations. Lambda behind Step Functions absorbs the burst and costs nothing between runs, which matters when the system is idle most of the day.
Trade-off Cold starts, and a hard execution ceiling per function. The workflow has to be decomposed into steps that each fit inside it rather than written as one long job.
Step Functions instead of orchestrating inside a single function
Retries, timeouts and partial failure become configuration rather than code, and a failed run can be inspected step by step instead of reconstructed from logs.
Trade-off The state machine is its own artefact to maintain, and reproducing a run locally is harder than executing a script.
Fan out concurrently, one unit of work per respondent
Respondents are independent, so throughput scales horizontally and one bad generation cannot poison the batch.
Trade-off Coherence has to be enforced inside each respondent up front, because there is no shared pass at the end to reconcile them.
EventBridge Scheduler for timed submission
Responses that all arrive in the same second look like exactly what they are. Spreading submission across a schedule is the difference between usable data and an obvious artefact.
Trade-off A run stops being a single request and response, so progress and failure need a reporting path of their own.
Where it landed
- Live and publicly usable at datafact.site.
- No servers, containers or queues to operate — the backend is entirely managed AWS services.
- Scales from a single respondent to a large batch without a configuration change.
In hindsight
Serverless made the scaling question disappear and replaced it with an orchestration question, which is a trade I would make again. What I underestimated was observability: with work spread across a state machine and many short-lived functions, you have to design how you will watch a run before you write the run itself.
Built with
- AWS Lambda
- AWS Step Functions
- API Gateway (HTTP)
- EventBridge Scheduler
- Generative AI
- Serverless
Next case study
NusaVerify
A fact-checker that scores how likely a claim is to be a hoax, and shows the entire reasoning chain behind the number.