0+
Years of experience
Lead Data Engineer · Data + AI · Cloud & Big Data
Engineering the data
foundations behind
smarter businesses.
Biography
Pallaav Sethi — a Lead Data Engineer building scalable data platforms, pipelines and AI-powered solutions across AWS, Azure and the modern data ecosystem.
I design the architecture and then build it: cloud data platforms, production pipelines and intelligent systems that turn raw data into decisions businesses can trust.
Years of experience
0+
NUMBER OF PROJECTS DELIVERED
0
Platforms I build with
'%20d='M126.916.072c-64.832%200-60.784%2028.115-60.784%2028.115l.072%2029.128h61.868v8.745H41.631S.145%2061.355.145%20126.77c0%2065.417%2036.21%2063.097%2036.21%2063.097h21.61v-30.356s-1.165-36.21%2035.632-36.21h61.362s34.475.557%2034.475-33.319V33.97S194.67.072%20126.916.072M92.802%2019.66a11.12%2011.12%200%200%201%2011.13%2011.13a11.12%2011.12%200%200%201-11.13%2011.13a11.12%2011.12%200%200%201-11.13-11.13a11.12%2011.12%200%200%201%2011.13-11.13'/%3e%3cpath%20fill='url(%23SVGlXoRkc4T)'%20d='M128.757%20254.126c64.832%200%2060.784-28.115%2060.784-28.115l-.072-29.127H127.6v-8.745h86.441s41.486%204.705%2041.486-60.712c0-65.416-36.21-63.096-36.21-63.096h-21.61v30.355s1.165%2036.21-35.632%2036.21h-61.362s-34.475-.557-34.475%2033.32v56.013s-5.235%2033.897%2062.518%2033.897m34.114-19.586a11.12%2011.12%200%200%201-11.13-11.13a11.12%2011.12%200%200%201%2011.13-11.131a11.12%2011.12%200%200%201%2011.13%2011.13a11.12%2011.12%200%200%201-11.13%2011.13'/%3e%3c/svg%3e)
Impact
Numbers from real delivery
0%
Pipeline processing time reduced
0%
Cloud & data cost reduced
0%
Reduction in data errors
0%
MoM sales performance uplift
About
Engineering systems that scale.
I work where architecture, engineering and AI meet — across financial services, enterprise data platforms, fraud and risk management, cloud modernisation, governance and data warehousing.
My work spans the full lifecycle of a data platform: understanding the business problem, designing the architecture, engineering the pipelines, running it on cloud, layering AI on top and then optimising for performance and cost. I have built fraud detection pipelines, enterprise finance foundations, credit and bureau data marts and an AI-powered sales assistant running in production.
B.Tech, Computer Science & Engineering — SRM Institute of Science and Technology, Chennai (2021), with a minors in AI/ML and Data.
Data platform blueprint
From raw signals to decisions
Sources
Pipelines
Cloud
Analytics
AI
Reliable
Governed
Scalable
Expertise
A technology ecosystem, not a checklist.
Data Engineering
PythonSQLPySparkApache SparkETL / ELTData PipelinesData ModellingData Warehousing
Cloud
AWSAzureADLSS3LambdaEMRGlueRedshiftKinesisAthenaStep FunctionsAPI Gateway
Modern Data Stack
DatabricksSnowflakeDelta LakeApache IcebergMicrosoft FabricUnity Catalog
Streaming & Distributed
KafkaSpark StreamingChange Data CaptureEvent-drivenReal-timeBatch
Data Architecture
Data LakeLakehouseMedallionDistributed SystemsSystem DesignLow-latency
AI & Intelligent Systems
Generative AIAI-powered applicationsVector DatabasesIntelligent AutomationBehavioural Analytics
Services
What I Can Build For You
01
Modern Data Platforms
Data lake and lakehouse platforms on cloud, with warehouses, modelling and ETL/ELT designed for both batch and streaming workloads.
Data lake / lakehouseCloud data platformsWarehousing & modellingBatch + streaming
02
Data Pipeline Engineering
Reliable ETL/ELT pipelines that integrate APIs and enterprise sources, with quality checks and performance tuning built in.
API & source integrationStreaming and CDCData quality checksPipeline optimisation
03
Data + AI Solutions
AI-powered internal tools, assistants and automated workflows grounded in your own data — like ARYA, an AI sales assistant I built in production.
AI assistants & internal toolsData-driven automationVector databasesAI/data infrastructure
04
Cloud & Data Modernisation
AWS and Azure architecture, migration off legacy systems, and performance work on the infrastructure your data runs on.
AWS / Azure architectureMigration & modernisationPerformance engineeringCloud infrastructure
05
Data Quality & Governance
Governance frameworks, validation rules and data models that make reporting trustworthy and audits straightforward.
Governance frameworksValidation & qualityRobust data modelsCompliance controls
06
Data Platform Cost Optimisation
Reduce platform spend without sacrificing performance — a 60% reduction across AWS and Snowflake on a live enterprise estate.
Spend analysisWorkload right-sizingQuery & storage tuningSustained savings
Experience
Where I have built.
Sep 2025 — Present
Senior Data Engineer — BP
Building fraud and risk data pipelines across Azure Data Factory, Databricks and ADLS, automating fraud detection with Auto Loader-based ingestion and contributing to the Finance Data Foundation and wider cloud modernisation.
Jul 2022 — Aug 2025
Data Engineer — Piramal Finance
Delivered Credit and Bureau DataMarts on Snowflake from 10+ source systems with governance controls, built AWS Glue pipelines and the Credit Scrub process, drove a 40% cloud cost reduction and engineered ARYA, an AI-powered sales assistant with behavioural analytics.
Nov 2021 — Jun 2022
Software Engineer — Toyota Connected
Built real-time ingestion and analytics for vehicle performance data using Java microservices, Lambda, DynamoDB and API Gateway, with infrastructure as code and CI/CD pipelines.
Jun 2021 — Oct 2021
Technology Consultant — EY
Delivered IT audits, information security and DevOps compliance reviews, automating risk assessment work in Python alongside business stakeholders.
Selected Work
Selected Case Studies
How I Work
Four steps, no mystery.
01
Discover
Understand the business problem, the existing architecture and the real state of your data landscape.
02
Architect
Design a scalable, cost-conscious solution with a clear delivery path and trade-offs made explicit.
03
Build
Engineer production-ready pipelines, platforms and AI/data systems, hands-on and in reviewable increments.
04
Optimise
Measure performance, reliability, data quality and business impact — then tighten what matters.
Certifications
Credentials behind the work.
FeaturedSolutions Architect – Associate
AWS Certified
FeaturedGenerative AI
Databricks
Data Engineer – Associate
MongoDB
Data Analytics Professional Certificate
Python Data Structures
University of Michigan
Forward Deployed Program
McKinsey & Company
DPDP for Engineers
Digital Privacy India
Have a data problem worth solving?
Whether you are building your first data platform, modernising an ageing stack or turning your data into AI products, I can help you design it and build it.
Contact