Ethan Aaron’s Post

Automated dashboards vs spreadsheets. Which one should you choose? __ The biggest benefits of automated dashboards are also their Achilles heel. I think a LOT about the differences between spreadsheets (Excel, Google Sheets) and modern data visualization tools (and all the stuff that goes into powering them - warehouses, ELT, etc). __ I started my career at Goldman Sachs. I was the spreadsheet expert for my investment team. When we needed to do complex stuff. When we needed to simplify a web of dependencies. I was the person digging into things. I even hacked together some VBA every once in a while. When you spend 100 hours a week for enough weeks, you realize that once you get data into a spreadsheet, almost any analysis is possible. __ It wasn't until I joined a data startup and got started on my path to 10,000 hours with SQL that I realized the 2 core limitations of spreadsheets: 1. Scalability 2. Automation I immediately fell in love with automated data pipelines. I started telling everyone in finance that spreadsheets were a thing of the past. With modern data infrastructure you could do everything you could do in a spreadsheet. But with modern infrastructure you could also: 1. Run analysis at crazy large scale 2. Keep your entire analysis up-to-data AUTOMATICALLY Like most people in the data world, I looked around and thought we were the most powerful people in the company because we could analyze anything, at scale, automatically. __ A few years later, I've come to realize that those two assets of modern data infrastructure are also their largest liabilities. The scalability of cloud infrastructure means you can run up CRAZY cloud costs in almost no time. And the fact that everything is automated sounds nice at face value, but it's also scary to think about how dependencies work. __ Think about this. When someone creates a new spreadsheet, NO upstream dependencies are created. They're just analyzing some data they have. But, when someone created a dashboard (automated or not), there is either a: 1. Hard dependency created with upstream teams and systems, or a 2. Possible dependency of the user ever tries to refresh the data As more users have access to spreadsheets, they're not creating inherent upstream dependencies every time they do something. But as more users start creating dashboards in your data infrastructure, you'll end up with a web of crazy dependencies. __ There are different ways to try and tame this complexity: 1. Data contracts 2. Capping the number of dashboards 3. Minimizing access to visualization tools 4. Buy even more tools I recommend a combination of 2 and 3 above (unless you end up with truly production workflow, then I'd recommend punting it to your engineering team to own the solution). What do you think? Are these implied dependencies as scary as I think they are? Or am I crazy... (Every dependency implies storage, compute, ingestion, and human capital -- that's scary to me)

Shadow data teams > self-service analytics Why? Two reasons: - The accountability - The cost structure __ 1. The accountability For some reason data teams love the idea of 'self-service analytics' even though I've never heard of it successfully creating business value in a company (except the largest scale tech companies in the world). I think technical teams must love the idea because they're no longer accountable for anything except a data platform. They're no longer accountable for ANYTHING that impacts the business. No accountability for revenue. No accountability for reducing costs. They can just focus on technology. In my opinion, that's an awful scenario for the company. What's the alternative? A data function that takes on fewer things (the really high value things), and feels ownership over solving those problems. By saying no to: 1) low-value stuff, and 2) ad hoc requests, you're bound to end up with shadow data functions and technology. As long as your procurement team can manage the security and the costs of those technical decisions, let it happen. Those teams can be accountable for solving problems for their business unit, and the centralized data team can be accountable for the highest priority problems for the company. 2. The cost structure For data teams, fixed costs are bad. Recurring headcount, recurring platform fees, recurring ANYTHING. You want costs that increase and decrease in a way that correlates to the value they create for the business. When you undergo a massive 'self-service' initiative, you're effectively saying - 'I want to invest a bunch of money into R&D, so someone else can try and create business value later one'. You have to pay for the development. And you have to pay for all the ongoing stuff associated with your fancy new self-service technology. And it's really really difficult to throw things away or pivot as the needs of the business change. (And the needs of the business ALWAYS change) Alternatively, if you stay lean, and focus on fewer things. The things that really matter to the business. You can see a problem and spend money to solve it. If the problem goes away, you can turn off that part of your tech stack and stop spending the money again. Lower fixed costs, higher flexibility, and higher ROI. You don't end up with high recurring bills just to keep the lights on. The people who are running 'shadow data teams' are making similar decisions in their own business unit. A smart CMO isn't going to allow for a massive R&D project if there's not path to ROI. An ad-hoc marketing analytics initiative is a great use case for shadow data work. Be aware of what's going on, but let it happen. __ In really really large companies things change. Everything changes. But most companies are not: - Netflix - Airbnb - Uber - Etc. And they should 1) embrace shadow data functions, and 2) be extremely wary of high R&D self-service data initiatives.

Data teams over the last 5 years - 2020 --> No data team - 2021 --> 1 person data team - 2022 --> 3 person data team - 2023 --> 20 person data team 🤯 - 2024 -- > 3 person data team I've been seeing this trend pretty consistently across the board with companies I talk to. __ I'll skip over the history of what happened and why, and instead talk about what you do now that you have a small team remaining after all of this happened. __ First, you need to dramatically reduce surface area to something that can actually be managed by a small team. This means getting rid of A LOT of stuff. I'm helping a company think through this right now. Start with the people surface area. i.e. access to BI and visualization tools. Go to teams (finance, marketing, HR, etc) and tell them you're drastically reducing the surface area your team can manage and you are REMOVING their team's access to BI + data visualization tooling entirely. Start with leaders in those teams (the CFO, CMO, etc). Then have them email their entire team and cc you. When you say 'We're entirely removing your access to {{ visualization tool }}', you'll see if anyone gets REALLY upset about the impact. Take note of who is REALLY upset and why they're upset, and which business workflow they're powering from existing data assets. Those are great indicators it might be a really high value data asset. You'll be shocked by how many teams and people don't actually care about ANYTHING your data team has built. From there, turn off everyone's access that isn't using the visualization tool for a mission critical workflow. I'd expect to cut user access by 50-75%. Hopefully more. - You'll save money on your viz tool license - You'll be able to remove unneeded transforms - You'll reduce your warehouse compute bill - You'll be able to shut down ingestion flows - You'll drastically reduce complexity I always recommend cutting VERY deep early on. You can always add things back over time if they become mission critical. But when 1-3 people are trying to maintain a system + team that used to be managed by 20 people, you need a very hard reset of expectations and data culture across the company. (And by the way, if everyone in the company is fine with you turning off their access, there's probably some work to do on how you prioritize what you build and maintain as a data team)

The partnerships team at Portable has been hard at work! And we're ecstatic to announce that we have achieved Google Cloud Ready - BigQuery designation. Why does this matter? 1. BigQuery is awesome (we use it internally) 2. Portable will have opportunities to collaborate closely with Google’s teams to help analytics teams drive business value 3. You can be more confident than ever to try out Portable's ETL solution There are four scenarios where data professionals engage Portable to help with BigQuery ETL: 1. Standing up a data stack. It’s tough to try and find the perfect fit from an ever-growing list of ETL tools. With a vast network, and first-hand knowledge of standing up modern data infrastructure, the Portable team is excited to help. 2. Ingesting data from niche applications. With 1000+ integrations, and lightning-fast development capabilities, Portable empowers teams to ingest data from common and long-tail data sources. All from a single platform. 3. Saving money on data infrastructure. Portable offers fixed rate pricing and a comprehensive suite of connectors so teams can spend less time worrying about pricing, and more time focused on generating business value. 4. Looking for white-glove support. In the ETL world, APIs change, schemas evolve, and on-call issues are constant. Sometimes, data teams just need a trusted partner that will smile, overly communicate, and treat you with respect. __ Super proud of our team for all the hard work! More exciting announcements to come. https://lnkd.in/eG8ZKFbN

Before you design a dashboard Before you write a line of SQL Before you think about the takeaways and the narrative Before you show someone the deliverable __ Ask yourself three questions: 1. Who is this for? 2. What cadence are they going to look at this? 3. Why? __ Make sure you only have one answer to each of the questions. Never try and build one thing for everyone and every use case. If the dashboard is for: 1) The CEO 2) For a weekly review of company KPIs 3) So that the company can measure progress towards goals (rewarding success, and providing ways to close gaps when they exist) Then make sure every single thing on the dashboard is... For the CEO In service of a weekly review of company KPIs Helps the company measure progress against goals (rewarding success, and providing ways to close gaps when they exist) __ Remove everything else.

As a data team, it's easy to be blindsided with bad news. You spend years architecting, designing, building, and then... Leaders question why your team exists. Where's the value? __ So, how do you make sure your data team's priorities align with the priorities of the company? Here's a trick for you. Find someone in the business that you trust. It could be a marketer, and HR professional, someone on the sales team... And ask them to do a 360 degree feedback cycle for the work your team is doing. - Give them a list of 10 business stakeholders that you believe see value in the work you do - Have them ask questions like: Which data assets do you use? How frequency do you use them? How would you quantify the impact of those assets? How could the data team have a larger impact? Is there anything the data team built for you that you don't need? - Ask them to take detailed notes. If it's easiest to get honest feedback, have them only share anonymous feedback It's not that large of an ask. And it gives you genuine feedback on how others in the company view the impact of your team. __ To your face, it's difficult for people to say you're doing great... But in a 360 review cycle, you'll likely get more honest feedback. __ You'll realize that the work you think is valuable might not be as valuable as you thought. And you'll catch issues before you get blindsided.

The really important things in data have NOTHING to do with data: 1. How to say no to low value stuff 2. How to read a P&L 3. Storytelling and how to craft a narrative 4. Nothing matters except business value 5. Business people don't care about SQL 6. How to calculate ROI 7. How to create a positive ROI 8. How to market things to stakeholders 9. How to get stuff done __ 10 out of 10 times I would... Take a smart person that's great at 1-9 above and invest in teaching them architecture, tooling, and SQL. It's a lot easier than trying to teach a techie why business value matters.

Want to learn best practices for dashboard development? I'm chatting with JJ Reynolds tomorrow and sharing strategies, tips and tricks for building actionable dashboards that move the needle for your business. __ Learn more and sign up in the comments! And trust me, it'll be actionable advice.

The responsibilities of a head of data have fundamentally changed in the last 3 years. (Well, the optics have at least) Three years ago, every head of data was: 1) Saying how data would be the most important team for the company (sounded a lot like AI today) 2) Accumulating tools and technology to power this visionary future 3) Hiring a massive team to harness the tooling to accomplish this visionary future. Two years ago (almost exactly) rates started going up, valuations came down, and budgets for 'innovation' were deprioritized. The head of data's job became: 1) Downsizing the team and communicating why it was happening 2) Starting to identify how the team was creating tangible business value (increased revenue or reduced costs) 3) Removing tools that were an obvious waste of money This last year has been the most interesting in my opinion. Teams were already downsized, and we were entering a phase where the head of data was tasked with: 1) Continuing to think about ROI (you need to prove the value you can generate from every dollar you're investing) 2) Digging deeper into cost savings, but beyond the low hanging fruit - this means tough decisions around deprecating hundreds of dashboards, or migrating from an expensive ELT tool to a more cost effective solution like Portable 3) Slowly rebuilding a team of generalists that can get their hands dirty with business problems __ The head of data is no longer a technical role. It's a P&L manager. It's an executive who can take high impact initiatives for the company and fully own them to completion. They just happen to be equipped with the knowledge and expertise of using data to get stuff done.

2024 is the year of NO NEW FEATURES in the data world. The last few years we saw a bunch of new data companies appear. They were all trying to invent new stuff: - Reverse ETL - CI/CD for data - Data tooling for your data tooling Coming off the heels of Data Universe this week, I came to realize that in reality, not a lot has changed over the last few years. > Reverse ETL companies have become CDPs (which have existed for a long time). They can pull from data warehouses as a source (but so can CDPs...) > We're seeing a new wave of data visualization tools that look A LOT like the last wave of data visualization tools (the ones that were acquired by big tech companies) > ETL + ELT is simply game of 1) reliability, 2) customer support, 3) connectivity, and 4) cost. As the cost of reliable solutions increases, new companies like Portable can compete on reliability, customer support, connectivity and cost. (But it's still the same underlying product as ELT from 5 years ago) > The offering Matillion had for data transformation 5 years ago is honestly not THAT different than the data transformation solutions today... > Data governance is a people problem, not a technical problem > Behavioral data collection is still powered by companies and frameworks with a long history in the space > Streaming data and CDC still are synonymous with Debezium + Kafka > Consultants are still solving the same problems. Helping non-technical business people to answer basic questions with data, like 'How many customers do we have?' __ This year's technical innovation is going to be boring. We're going to see a lot of announcements that simply aren't exciting. __ In my opinion, that's a great thing. Because when there are no shiny things to chase, you have to find something else to do. We're going to see data teams: 1) Get really efficient at using those building blocks in a cost effective way 2) Stop chasing the latest new product or feature - and instead - digging deeply into business problems and ways to create value __ As I said, 2024 is the year of no new features.