Executive Summary

New Engen manages digital advertising for dozens of brands. Every week, they oversee $12-14 million in ad spending across platforms like Google, Facebook, and TikTok. Their success depends on giving clients clear, accurate reports about where their money goes and what results they get.

But in late 2023, New Engen faced a problem. Their team spent hours each week pulling data from different systems, checking numbers by hand, and building custom reports for each client. As they grew and added more clients, this manual work became impossible to manage. They needed a better way.

Soli & Co built New Engen a modern data platform using Google Cloud, dbt, and Airflow. The new system automatically collects advertising data from multiple sources, processes it, and delivers reports to clients. The platform handles multiple clients at once and grows with the business.

The results speak for themselves. New Engen's analytics team now spends their time finding insights instead of copying data between spreadsheets. The company can take on new clients without adding more people to handle reporting. And clients get faster, more accurate information about their advertising performance.

‍

Key Metrics

‍

• Weekly Ad Spend Managed: $12-14 million across all clients and platforms
• Client Scale: Dozens of brands with multi-tenant architecture
• Processing Efficiency: Incremental data processing reduces costs and speeds up refresh times
• Time Savings: Analytics team shifted from manual data work to strategic analysis
• Scalability: New clients onboarded through configuration, not custom development
• Data Sources: Multiple advertising platforms (Google Ads, Facebook, TikTok, etc.) unified in single warehouse

‍

The Challenge: Managing Millions in Ad Spend Without the Right Tools

‍

New Engen operates in the digital advertising world. They run campaigns across Google Ads, Facebook, TikTok, and other platforms for dozens of brands. Some clients spend over $1.5 million in just two months. The stakes are high, and clients demand detailed reporting on where every dollar goes.

Brandon Nelson serves as Director of Business Analytics and Reporting at New Engen. In November 2023, he faced a growing problem. His team managed data and reports for an expanding client base, but their tools couldn't keep up.

"We run campaigns across multiple platforms", Brandon explained during an early meeting. "They all have different APIs and data schemas. The question became: how do you design a process to harmonize that data so it can be used for analytics and reporting?"

‍

The Manual Reporting Trap

‍

The challenge went beyond just technical complexity. New Engen's team spent significant time on manual data work. They pulled numbers from various platforms, checked them against each other, and built custom reports for different clients. Each new client meant more manual work.

As the company brought on new accounts, this approach created serious problems:

Time and Resources: Analytics staff spent hours on repetitive data tasks instead of finding insights that help clients.

Scalability: Adding new clients meant adding more manual work. The team couldn't grow revenue without growing headcount at the same rate.

Consistency: With manual processes, keeping data accurate across different reports took constant checking and rechecking.

Client Experience: Custom reporting for each client meant slower turnaround times and less consistency in what clients received.

‍

Technical Debt Slowing Down Innovation

‍

New Engen had built their initial data systems quickly to meet immediate needs. This created technical debt that made changes difficult. The company used BigQuery as their data warehouse, but the systems around it were built for a smaller operation.

Steve Tazic, VP of Technology at New Engen, understood that technology and process needed to work together. But the existing infrastructure made it hard to implement new ideas or scale operations efficiently.

The team faced a choice: continue with manual processes that worked but didn't scale, or invest in building a platform that could grow with the business.

‍

The Solution: Building a Modern Data Platform

‍

In late 2023, New Engen partnered with Soli & Co to redesign their data infrastructure. The engagement focused on building a scalable, automated system that could handle multiple clients and grow with the business.

The approach centered on using proven, market-standard tools rather than custom solutions. This strategy meant New Engen would benefit from tools that other companies actively maintain and improve, reducing long-term maintenance costs.

‍

The Technical Foundation

‍

Soli & Co built the new platform on Google Cloud Platform, which New Engen already used. The architecture focused on three key components:

Data Warehouse (Google BigQuery): All client data flows into BigQuery, Google's cloud data warehouse. This gives the team a single place to store and query data from all advertising platforms.

Data Transformation (dbt): Instead of writing custom code for every data transformation, the team uses dbt (data build tool). This open-source tool lets analysts write transformations in SQL and manage them like software code. Changes get tested before going live, and the team can track what changed over time.

Orchestration (Airflow): Apache Airflow schedules and monitors all data processes. It ensures data gets pulled from advertising platforms on time, transformations run in the right order, and the team knows immediately if something breaks.

‍

Designing for Multiple Clients

‍

One of the biggest technical challenges was building a system that handles many clients efficiently. The team needed to avoid building custom processes for each new client.

The solution used a multi-tenant approach. The data platform treats each client as a configuration, not a separate system. Adding a new client means updating a configuration file, not writing new code.

"The incremental setup of this is the tricky part," Brandon noted during implementation discussions. The team spent significant time getting the incremental data processing right so the system only processes new data instead of reprocessing everything each time.

‍

Building for Reliability and Cost Efficiency

‍

The platform needed to run reliably while keeping costs under control. BigQuery charges based on how much data you process, so inefficient queries can get expensive quickly.

The team implemented incremental data processing. Instead of reloading all historical data every time, the system identifies and processes only new or changed data. This approach dramatically reduces processing costs and speeds up data refresh times.

They also built monitoring into every step. When something goes wrong, the team knows immediately and can fix it before clients notice.

‍

Implementation: From Concept to Production

‍

Soli & Co and New Engen took a methodical approach to implementation. Rather than trying to rebuild everything at once, they worked in phases.

The first phase focused on getting the core infrastructure right. This meant setting up BigQuery, configuring dbt, and ensuring Airflow could orchestrate the data pipelines reliably.

Next came migrating client data. The team started with a single client to prove the approach worked, then gradually moved more clients to the new platform.

Throughout implementation, Brandon and the Soli & Co team met weekly to review progress, solve problems, and adjust the approach based on what they learned.

‍

Solving Real-World Problems

‍

Building a data platform involves solving many small problems that add up to big challenges. The team worked through issues around data quality, API rate limits, schema changes, and more.

One key challenge involved handling different data update frequencies. Some advertising platforms provide data in near real-time, while others have delays. The system needed to account for these differences and still deliver consistent reports to clients.

Another challenge centered on handling client-specific requirements. While the platform uses a standard approach, different clients have different reporting needs. The team built flexibility into the system without sacrificing the benefits of standardization.

‍

Knowledge Transfer and Documentation

‍

A critical part of implementation involved ensuring New Engen's team could operate and maintain the platform independently. Soli & Co focused on knowledge transfer throughout the engagement.

"I spent a lot of time establishing processes, tools, monitoring, checking, and ways of operating," Brandon explained. "We created wikis and guides and all sorts of resources."

This documentation became essential as New Engen brought on additional team members who needed to understand how the platform works.

‍

Results: From Bottleneck to Business Advantage

‍

The new platform transformed how New Engen's analytics team spends their time. Instead of manually pulling and checking data, automated processes handle routine tasks.

The team can now focus on higher-value work: analyzing trends, finding optimization opportunities, and providing strategic advice to clients.

‍

Scalability Without Adding Headcount

‍

Perhaps the most significant business impact comes from scalability. New Engen can now take on new clients without proportionally increasing their analytics headcount.

Adding a new client to the platform takes configuration changes, not months of custom development. This changes the economics of growth for the company.

‍

Improved Client Experience

‍

Clients benefit from faster, more consistent reporting. The automated platform delivers reports on schedule without manual intervention. Data is more accurate because it flows through standardized processes with built-in quality checks.

For clients spending over a million dollars per month on advertising, this reliability matters. They need confidence that their performance data is accurate and timely.

‍

Technical Benefits

‍

From a technical perspective, the platform delivers several advantages:

Cost Efficiency: Incremental data processing and optimized queries keep BigQuery costs under control even as data volume grows.

Reliability: Automated monitoring catches problems quickly. The team can often fix issues before they affect client reporting.

Maintainability: Using standard tools like dbt and Airflow means the team can hire engineers who already know these systems, reducing training time and making the platform easier to maintain.

Flexibility: The platform can adapt to new requirements without major rewrites. When New Engen needs to add a new data source or report type, the existing infrastructure accommodates it.

‍

Setting Up for Future Growth

‍

The platform positions New Engen for continued growth. As they add more clients and expand into new advertising platforms, the infrastructure can handle the increased load.

The company also has a foundation for more advanced analytics. With clean, reliable data flowing through the platform, they can build machine learning models, implement automated alerting, and create more sophisticated optimization strategies.

‍

Key Lessons: What Made This Work

‍

One of the most important decisions was using established, market-standard tools rather than building everything custom. This approach provided several benefits:

Reduced Risk: These tools have been tested by thousands of companies and have active communities fixing bugs and adding features.

Easier Hiring: Engineers familiar with dbt, Airflow, and BigQuery are easier to find than specialists in custom systems.

Long-term Viability: These tools will continue to be maintained and improved by their communities and sponsors.

‍

Process and Technology Together

‍

Steve Tazic's understanding that technology and process must work together proved essential. The best technical platform fails without good processes around it.

New Engen invested in documentation, training, and establishing clear procedures for operating the platform. This investment paid off as team members could work effectively with the new system.

‍

Incremental Progress Over Big Bang Changes

‍

The phased approach reduced risk and allowed for learning along the way. Rather than trying to migrate everything at once, the team proved concepts with one client before expanding.

This approach also kept the business running smoothly during the transition. Clients continued receiving reports without disruption while the new platform came online in the background.

‍

Partnership and Communication

‍

The weekly deep-dive meetings between Brandon and the Soli & Co team created a strong foundation for success. Regular communication meant problems got identified and solved quickly.

This partnership approach, rather than a traditional vendor relationship, led to better outcomes. Both teams brought expertise and worked together to find the best solutions.

‍

Looking Forward

‍

New Engen now has a data platform that can grow with their business. As they continue expanding their client base and adding new advertising platforms, the infrastructure can handle it.

The platform also opens up new possibilities. With reliable, clean data and automated processes, New Engen can focus on advanced analytics and machine learning to provide even more value to clients.

For companies managing significant advertising spend across multiple platforms, the lesson is clear: investing in solid data infrastructure pays off. The time and resources spent building a scalable platform create lasting competitive advantages.

New Engen transformed their data operations from a constraint on growth into an enabler of it. They can now scale their business without scaling their problems.