Where I've built
AI Data Engineer
Nov 2024 — Present- Architected enterprise-scale Azure Lakehouse (ADLS Gen2, Delta Lake, Databricks, Synapse) processing 5+ TB daily across $500M+ annual transaction ecosystems
- Reduced data latency 65%, reporting cycle from 48h to under 6h
- Real-time fraud detection with Event Hubs + Spark Structured Streaming monitoring $500M+ transactions — threat response decreased 70%
- AI recommendation systems (PyTorch, Azure ML, MLflow) influencing 1M+ interactions, +22% engagement, -40% model deployment cycles
- Delta Lake partitioning & Z-order indexing — BI performance +55%
- Terraform IaC with PCI-DSS/SOC2 compliance
Software Data Engineer
Jan 2021 — Jul 2023- Architected AWS Glue + PySpark ETL processing 100M+ financial transactions annually — 99.9% accuracy, 45% faster, 30% cost reduction
- Designed Redshift warehouse integrating 15+ enterprise systems, delivering $120K+ annual savings
- Serverless ingestion (Lambda, S3, Step Functions) eliminating 500+ manual hours at 99.95% SLA
- Real-time monitoring with Kinesis + Spark Streaming on $400M+ portfolio assets, fraud response -35%
- SageMaker anomaly detection improving fraud precision 18%
- Terraform IaC across multi-environment architectures
Technical Graph
Selected Works
Rust HFT Risk Engine
Real-time risk engine streaming synthetic venue data with pre-trade credit and price collar checks. PTP timestamped to ClickHouse with deterministic replay.
Latent Diffusion Engine
Latent-diffusion pipeline with point-in-time content embeddings trained on 10 H100 nodes with Artbench + OpenImage data.
Notion Data Spine Clone
CRDT collaborative editing on Kafka, Postgres, and gRPC with backpressure handling and idempotent writes.
HPC Image Captioning
Memory-efficient DDP with CNN-RNN, crash recovery via gradient checkpointing across DenseNet169, InceptionV3, ResNet50.
How I see
Credentials
Certifications
Enterprise multi-cloud architecture design across AWS, Azure, and OCI platforms.
Verify ↗Cloud database design, migration patterns, and autonomous database management.
Verify ↗Machine learning model lifecycle, feature stores, and model deployment on OCI.
Verify ↗LLM integration, RAG architecture, and prompt engineering on Oracle Cloud.
Verify ↗GPU-accelerated LLM inference, TensorRT optimization, and multi-node training.
Verify ↗MLflow lifecycle, feature engineering, AutoML, and production MLOps on Unity Catalog.
Verify ↗Strategic problem-solving, structured communication, and leadership development.
Verify ↗Core AWS services, security, architecture, and cloud economics fundamentals.
Verify ↗SageMaker model training, deployment pipelines, and ML infrastructure on AWS.
Verify ↗Education
Generative AI, HPC with Deep Learning, Big Data & Indexing. Co-founder at CareWallet.
Probability & Statistics, Cloud Computing. 2× GCP Quest Leader. Google Code-in Mentor.
Contributions
Word of mouth
"Shrey is rare in that he combines strong engineering depth with clear product thinking. He's thoughtful about schemas, data quality, and monitoring..."
"He managed end-to-end data workflows — handling everything from source extraction and complex transformations to delivering well-structured data marts."
"His proactive stance on identifying and mitigating potential challenges underscores his critical role in our risk management strategies."
"Shrey offered insights that added strength to our technological foundation, paving the way for operations that scale."
Let's talk
Or let Spectra set up a call (Voice coming soon)
Or reach me directly: patelshrey77@gmail.com