If you are thinking, “Aaaargh, why are we keeping all this data in a zillion different places? I can’t find anything!” or “They lost/erased that file again… How am I supposed to make that report now?” or “Why is this data here? No one needs it!“ or “This is good, but it could be way better.”, you are probably in a pretty good place to change your process for handling the data.
1. Data synchronization in a nutshell
Data synchronization is an essential aspect of data management that helps businesses to manage and maintain their data effectively. Unfortunately, many companies need help handling their data, which can lead to errors, delays, and even lost data.
We will provide a step-by-step guide to starting a data synchronization solution and explore the key considerations and challenges involved. By the end of this article, readers will have a clear understanding of data synchronization and will be able to implement it successfully within their organizations.
Still trying to figure out where to start? Check out this template and answer the following questions.
2. Define your process
Look up the Business Requirements sheet.
2.1. Why is it required?
Identify all the problematic areas and how they affect your work. For example, is it practicality, speed, sustainability, security, etc.? How would you benefit from changing the process? This will help you understand the current process state and whether the change is worth your while.
2.2. What is required?
Think about how the ideal process would look from your point of view. Which datasets and data points do you need, and what does it represent? How do you want to use the data now and in the future? This will be your goal and guide.
2.3. Who do I need?
Find out who designs and controls the current process, who provides you with the data, who maintains the technology currently used, who else uses the data, who else can benefit from this project, etc.
This will help you identify all the key business and technical stakeholders, subject matter experts, and users on which this project can and will depend.
3. Define your data
Look up the Data Requirements sheet.
3.1. Which data do I require?
Get together with the team and define in detail which datasets and data points you need. Besides the obvious ones, you might need some other data points. Think if you need:
- the data to be filtered by some parameter (for example, country)?
- time component to be included (for example, fiscal year)?
- additional data points often used along with your data (for example, you need sales data, but it is in local currency; you might also want to include conversion rates)?
- additional data points that give a better context for your data (for example, you need sales data on store level; it might be helpful to include store opening hours to understand the variations in sales data)?
- additional data points that are not needed now but might be helpful in the future (for example, you need weekly sales data, but one day you might want to look at the data daily)?
- historical data and for which period?
3.2. What is my data source?
The goal is to identify the data sources to get it in its most authentic and rawest format possible. Find out if a data source is trustworthy or intermediate and if it does some manipulations and transformations on the data. Understand precisely how and how often your data is collected and delivered to the data source. Check for any plans for the data source to change or be sunsetted. This will enable you to see any dependencies or limitations to your project.
3.3. How is my data interconnected?
Are some data points a result of the calculation of several other data points? Or is it a concatenation? Are some data points a sub-category of other data point? Can the data points from two datasets be linked? Understand how your data points are interconnected and, based on that, determine which ones you need, which ones you can derive from others, how they should be grouped, linked, etc.
3.4. How to map my data to new data points?
So you have identified all the data and connections needed. You are almost done with the mapping! Next, consolidate everything into a list that defines how each new data point is connected to the existing ones.
3.5. How can I ensure there are no duplicate data?
You need to define a primary key! A primary key is a data point or a combination of data points, making each data row unique. By filtering the data by the primary key when inserting new or changing the existing data, you ensure there will be no duplicates. Also, see if you can get the data in an automated way rather than manually entering it. If there is no way but a manual one, restrict the number of people who enter the data.
4. Define your technology
Look up the Technology Requirements sheet.
4.1. How do I transport the data?
So you are not lucky enough to have all the data in-house. Don’t worry. Each data source will be able to provide you with one or more (if you’re lucky!) options to pull the data. It may be an Excel file sent by e-mail, dropped on Sharepoint, or an API. So gather the tech teams from all sides, discuss your options, and understand and define the means of data transportation.
4.2. How does my process depend on other ones?
For example, your process needs to get fresh data from the data source twice a day. However, the data source updates the data only once daily since they need time to prepare it, and only at 1 PM. Obviously, you will only be able to pull the data once a day or at 11 AM since you will not get a fresh load. Therefore, you need to understand all the underlying processes, schedules, and restrictions to succeed.
4.3. What if there are issues after my process is in place?
The people who helped you this far might not be responsible for supporting the process later on. And you might need more time to deal with it! In case issues arise (and, unfortunately,they probably will), define the exact process of when the alert should be triggered, how it should look, and who should be alerted. Identify and determine the support chain on all included sides.
After reading this, we hope you are ready to start your company's data handling process. Good luck, and feel free to contact us for any additional information.
