Introduction
Back in the early 90s, companies and organizations started investing in what was referred to as a webmaster — essentially the person that was in charge of all things having to do with the internet. As website needs became more complex and core to operations, we realized the internet wasn’t something that could be handled by one webmaster. Something similar has happened with data, where organizations invest in a “Data Person” to oversee all things data. But just like with the internet, collecting, storing, and analyzing data requires lots of different skills and tools that can’t be found in a single person.
The Data Stack is essentially what moves your data from individual ingredients stored in individual systems, to one cohesive data environment that is accessible and usable. It is often referred to as Data Engineering. This guide will walk you through the four major parts of a data stack and point to some of the things you need to know and consider as you build out your data infrastructure.
In this Guide
- Ways to collect and share data in your organization.
- Recommended data storage technologies and solutions.
- Practices for keeping your data secure.
- Ways to strategically use data in your organization.
Data Collection and Sharing
The first part of your data stack is all about how you get data. Some organizations have extensive data collection tools and practices and that is how they primarily think about getting data. Data sharing can be a great way to get access to data as well. Oftentimes someone has the data that you are looking for and you are able to negotiate agreements to get access to that data (and share your own).
Some resources on Data Collection and Data Sharing:
Data Storage and Transformation
Once you have all of your data ingredients, it is time to get them organized and ready for consumption. This step could be considered a kind of data mise en place for those data-cooking nerds. This often is about moving data from a source system (such as your data collection tools or CRM) into a data warehouse where it can be accessed and analyzed more broadly and in conjunction with data from multiple sources. There are many vendors and tools that specialize in either ETL (Extract – Transform – Load) or ELT (Extract – Load – Transform) technologies that help move all of your data into one place, and several options for cloud providers who can store all of that data in one place.
Data Security
Don’t collect what you can’t protect is a useful motto to live by. Ensuring good data security tools and practices is an integral part of your data stack as an organization. Many social sector organizations are collecting sensitive data about vulnerable populations; we must do everything we can to ensure that we are protecting that information. This isn’t just about technology solutions, but also about training and equipping your staff to protect the data your organization collects. We often point to flaws in technology when data security issues arise, but it’s equally likely for human behavior to be the cause when staff is insufficiently trained, or targeted by social engineering.
Data Use
Once you have all of your data in one place and secure, you can start using analytics and business intelligence (BI) tools to turn your data into something valuable for your organization. There are many different options to consider when looking for BI & analytics solutions. A helpful frame is that the more technical skills people in your organization have, the more general the tools they can use. For example, if you have well-trained computer and data scientists on your team they’ll likely default to using tools like Python and R to interact with and manipulate large amounts of data. If your talent is more focused at the analyst level you will likely want to use tools like Tableau that are more visual in nature and more straightforward to use.
Conclusion
These resources will help you level up your organization’s data stack. This represents a maturation from a tool-based approach (i.e., I have and analyze my data in a set of tools) to take a more infrastructure-focused approach where you use tools more specifically to collect/ingest data, bring it together, and then analyze it from one place. As you invest more in this kind of capability, you are also increasing the value that you are able to get from your data. We also suggest reaching out to peer organizations, whose challenges and learning may provide direct and current examples of ways forward. Please feel free to suggest any other guides you found helpful by contacting us and we may incorporate them.
Join Our Community
Make connections with other social impact organizations and receive a curated listing of community and data.org events and opportunities.
By submitting your information and clicking “Submit”, you agree to the data.org Privacy Policy and Terms and Conditions, and to receive email communications from data.org.
Related Content
Instead of thinking in a structural, sustainable, long-term way about the biggest problems facing our world, we too often try to solve the world’s problems with new technologies implemented as quick fixes. Read more