Can Extracting Data for Request be Faster?

by Koo Ping Shung

One common scenario I  hear a lot is, "You are an expert in data right? Can you give me X broken down by Y for the past N years? Can I have it in 30 mins? Should be EASY for you right?"

So let me break this down for the layman, that request alone actually triggered the following questions in any data analyst or data scientists and anyone can be the judge of it if it is something that can be done EASILY and QUICKLY.

The questions are:
- Which datasets in the warehouse contain the needed data?
- How do I merge the datasets to get the data I want? What is the merging key?
- Is the data quality good in the datasets?
- How do I use the tools to create the final dataset? What is the SQL code to extract    the datasets?
- Did the SQL (or code or feature) work as expected?
- Did the datasets merge as expected?
- Did I manage to get all the data I want in the final dataset?
- Which visualization will be suitable for the requester?
- Why does he/she need it?
- How do I structure the data for the visualization?  
- What other questions will he/she want to get answered with the data?

The above are just a sample of the questions data professionals will ask themselves in order to prepare the numbers or report the business stakeholders need. So tell me is it really so simple and fast to extract information?
As you can see, it is not that straightforward, there is much thought into it, you can imagine the analyst or data scientist have these columns and datasets that are flying around in their mind and moving these parts together in their brain to come up with the final dataset that then can be pushed into the tools to crunch the numbers. It is really not easy.

So if you are someone who needs to consistently request for numbers to make the best decision possible. Make the job of an analyst or data scientist easier by letting them have the request as early as possible. :)

I hope you have enjoyed this post. I know my fellow professionals did when I published it on LinkedIn.

Do check out my other blog posts. Keep in touch on LinkedIn or Twitter, else subscribe to my newsletter to find out what I am thinking, doing or learning. :)


I've just started a contribution page at buymeacoffee.com. If I have been adding value to you, please consider making a donation to let me know I have made an impact. :)