Fraud Detection
A neural network that flags fraudulent credit card transactions, built on 1.85 million simulated purchases and served by a FastAPI service as a versioned 6 KB model bundle.
- Python
- TensorFlow
- ONNX
- FastAPI
- Redis
- Docker
Challenge
Only 0.52% of the 1.85 million transactions in the Sparkov dataset are fraud, so a model that never flags anything is 99.5% accurate. The model has to be judged on that rare class, and served the way a production model would be: versioned, validated, containerized and tested.
Approach
Split by time: train on 2019, validate on the first half of 2020 and keep the test set for one final evaluation. A small Keras network with two ReLU layers has its fraud class weight and training setup tuned on validation, three seeds each, and the decision threshold is the one with the lowest total cost, where a missed fraud costs its amount and a false alarm $10. Error analysis of 100 missed frauds and 100 false alarms showed both came from scoring each purchase on its own, so model 2.0.2 adds five features from the same card's previous 24 hours, computed only from its earlier purchases. Feature engineering is one package shared by training and the API.
Outcome
Card history raised PR-AUC from 0.855 to 0.966 (three seeds) and cut both error types at once: missed frauds from 204 to 56 and false alarms from 2,722 to 752, halving the cost on validation. Scored once on the test set, model 2.0.2 reached a PR-AUC of 0.959 and caught 95.8% of frauds with 69% precision. A test replays purchase sequences through the API and through the training code and checks they agree exactly. The data is simulated, so these scores would not transfer to real transactions.
Architecture
Decisions and trade-offs
Split by time, and score the test set only once
Training on the past and predicting the future mimics production. Every decision was taken on validation, which makes its numbers slightly optimistic, so the test set was scored once at the end and no model was changed after seeing it.
A fraud class weight of 10, not the class ratio
Searched on validation with three seeds: a weight of 10 reached a PR-AUC of 0.74, while the usual default, the class ratio (176), scored 0.55 and made the network flag far too much.
The decision threshold minimizes cost, not errors
A missed fraud costs the bank its purchase amount and a false alarm is assumed to cost $10. For model 1.0.2, the cheapest threshold, 0.23, cost 27% less than the default 0.5.
The threshold depends on that assumed $10: at $50 per false alarm it moves to 0.53.
The model ships as ONNX, with its preprocessing as JSON
ONNX Runtime runs it, so the API image (377 MB) needs neither TensorFlow nor scikit-learn, and loading a bundle never executes pickled code. Before publishing, the bundle is reloaded the way serving loads it and checked against the Keras predictions.
One feature package for training and serving
A transaction is always transformed the same way in both places, which avoids training-serving skew, where a model silently degrades in production because its inputs are computed differently from training.
Card history in Redis, one sorted set per card
The history is one range query, shared by every replica and kept across restarts, and /health answers 503 when Redis is down so a load balancer stops sending traffic.
A new deployment starts empty and sees every card as quiet until traffic arrives; it should first be warmed up from recent transactions.
Key evidence
PR-AUC of 0.959 on the untouched test set, catching 95.8% of frauds