Data Engineer Journey
Data Engineer at TP
When I first joined Tilting Point, I was part of a team comprising eight talented individuals: data scientists, analysts, managers, and a fellow data engineer. Together, we embarked on the journey of transforming raw data into actionable insights, creating a robust data pipeline that would power the company’s decision-making.
Our team maintained a complex Airflow environment on AWS, orchestrated with Kubernetes. We leveraged both Scala Spark and Python to develop sophisticated ETL jobs. Scala Spark handled the heavy lifting, processing large-scale data efficiently, while Python managed API calls and scrapers with precision.
This setup was crucial for our ETL processes, which integrated data from some of the biggest players in the space, including Apple, Microsoft, Steam, Google, Facebook Ads, TikTok Ads, Amplitude, Leanplum, and Xsolla.
A crucial part of our role involved translating stakeholder requirements into effective data pipelines. Through regular interactions with various business units, we ensured our data solutions aligned perfectly with their strategic objectives, often exceeding expectations.
The transformed data seamlessly flowed into Databricks Delta Lake, providing a solid foundation for reliable analytics. We then created real-time dashboards in Tableau, turning complex data into digestible insights. This empowered stakeholders across the organization to make informed, data-driven decisions.
2nd Project at TP: Data Platform Engineer | Software Engineer
Transitioning into the second stage of my journey at Tilting Point, I took on the role of the sole data engineer. My mission was to create a fully orchestrated platform within Databricks, leveraging Kafka and Spark Streaming to ingest data from our mobile gaming platform.
I implemented a real-time data pipeline using Kafka and Spark Streaming to handle live data ingestion. We adopted gRPC and Protocol Buffers (protobuf) to define the communication protocol between Kafka and various services within our gaming platform. This setup ensured low-latency, high-throughput data processing.
Though primarily focused on data engineering, my role required a multidisciplinary approach. I occasionally modified services written in Go to align them with our data requirements. For the data pipelines and Kafka integration, I utilized Python, capitalizing on its robust libraries and tools suited for big data processing.
Databricks became the cornerstone of our data platform. To maximize its potential, I participated in weekly workshop sessions with Databricks experts, honing my skills and exploring advanced features. This continuous learning process transformed me into a proficient Databricks professional, capable of leveraging its full capabilities.
Turning data into actionable insights remained a priority. I constructed visualizations using Lakehouse Monitoring, providing real-time dashboards that offered comprehensive visibility into the performance and health of our data pipelines.
In addition to building the platform, I developed an AI system using Databricks Genie Rooms, vector search databases, and Unity Catalog. This AI allowed stakeholders to directly query our ingested data or construct dashboards on-demand. By enabling natural language interactions and simplifying data access, we democratized data usage across the organization, making insights more accessible than ever.
Tech I used in Tilting Point
Throughout my journey at Tilting Point, I leveraged a diverse set of technologies to drive data excellence. Here’s a glimpse of the tools and platforms that powered our data initiatives. Numbers represent the frequency, 100 meaning every day.
Data Engineer | Machine Learning Engineer
Embarking on my data engineering journey at BBVA, I delved deep into the realms of Python and Scala, sculpting projects that would ultimately personalize and enhance the financial health of our clients.
One of the cornerstone projects in my tenure was the “Transaction Categorizer.” By harnessing the capabilities of Datio, Python, and Scala, we developed a robust system to categorize movements, be it transactions via cards, payrolls, transfers, or debts. This categorization was instrumental in painting a clearer picture of clients’ financial activities.
In parallel, I spearheaded the “Customer Personalization” project. This wasn’t just about crunching numbers; it was about crafting attributes that reflected the nuanced financial behavior of our clients. These attributes laid the groundwork for another project – “Financial Health” This study provided a comprehensive analysis of clients’ financial wellness, offering insights that were previously elusive.
But the journey didn’t stop there. Understanding that personalized solutions are the key to client satisfaction, I worked on a “Product Recommendation Engine” This engine, tailored to specific client segments, suggested products that matched their financial profiles, making personal financial growth more attainable.
Central to these projects was collaboration with our team of data scientists. They brought the magic with their machine learning models, and it was my responsibility to embed these models into our daily processes, ensuring seamless and efficient production. The stakes were high – working for a bank meant every deployment was critical, and any slip could ripple across the financial lives of our customers.
However, my proudest achievement remains leading with 2 more fellow engineers the implementation of a sophisticated data software framework. This framework not only simplified the deployment of AI models but also managed intricate data pipelines and orchestrated Spark-based applications. The result? A remarkable boost in our production speed and effectiveness.
My role at BBVA was more than just a job, it was a mission to transform data into actionable insights, ensuring every client walked their financial path with confidence and clarity.
Tech I used in BBVA
During my time at BBVA, I not only honed my social and negotiation skills essential for daily operations but also mastered a suite of technical tools and methodologies