10 Required Non-technical Skills for a Data Scientist

"Data Science(DS)" is nothing new but the term itself and the recent level of interest in it. As a practice it has commercially (not academically) existed for more than 25 years, mainly under "Data Mining (DM)" and "predictive analytics(PA)," since early 1990's. DM and PA got a lot of traction originally in financial, Telco, and retail industries that had a lot of granular historical data. Like anything that gets sudden attention and interest, DS has been misused and abused in a variety of ways. Given the fast surge in market demand in the last several years, many claim to be or want to be data scientists. True data scientists and DS managers who had to deal with screening DS resumes, can testify to the level of present noise (false positives) in that application process.

"Data Science" tries to be an umbrella field that covers more of what data mining and predictive analytics practices have covered. That is justified since with the growth of data of all kinds in recent decade and what is expected in the coming years, we need a lot more of the people with relevant DS skill sets. The challenge however has been the definition of that "skillset." What makes a good data scientist?

In my previous post "What is BuDAI?," I explained that a successful DS project requires the involvement of the data science team through the whole cycle. The core part of a data science project deliverable is the insight and decision coming out of analytics. The analytics could be trivial (generally aggregated view of data and only looking at a handful of variables together) where in that case there would be no need for DS. That would be in the realm of a data or business analyst. DS comes into picture usually where:

• More sophisticated analytics approaches are required,
• More complex transformations are required to prepare the data,
• Granular or atomic analysis of entities of interests is required,
• Analytics could be straightforward but big data is involved requiring attention to optimization of analytics,
• ...

Within BuDAI process, the S team has to interact with business, data engineers, data architects, project managers, and product managers to name a few. Aside from some relevant technical skills/knowledge[1] in math, stats, machine learning, programming, databases, and systems (the breadth and depth will depend on the level of seniority of the Data Scientist), through the years I have found the following ten traits to be as important as technical skills for junior hires and absolutely essential for senior data scientists.

1. Problem solving ability 
2. Business acumen
3. Ability to question the work of self and others,
4. Passion for data (the more data, the better)
5. Attention to details and ability to validate own work in multiple ways
6. Statistical thinking (a thinker who knows when to reason deterministically and when not)
7. Passion for exploration and discovery (quick learner from fails)
8. Ability to devise optimal ways to experiment new or creativity (finding novel useful insight is cumbersome. One can never find a sure way to find it)
9. Presentation ability (written and oral)
10. Ability to simplify complex concepts for explaining to others.

----------------------
[1] This is the subject of another blog and given the today's coverage of data science, the required technical abilities vary greatly.

---------------------------------------------------------------------------
I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

What is DS-BuDAI?

Data science[1] (covering data mining and related practices) is a multidisciplinary field that requires knowledge of a number of different skills, practices, and technologies, including but not limited to machine learning, pattern recognition, mathematics, programming, algorithms, statistics, and databases. In the context of big data, more skills and knowledge is required, such as knowledge of distributed computing techniques/algorithms and architectures. By nature, data science is a creative process that is a combination of both science, engineering, and art. Hence its success has been more dependent on the quality and the experience of the team that has been carrying it out. Thus in the past, for some time, data mining projects were not repeatable with the same level of success across different enterprises. However, with the maturity of the practice, that has changed.

Since the late 1990s there have been a variety of efforts to create standard methodologies and process models for data mining, such as CRISP-DM (Wirth and Hipp 2000). In this methodology there is an important focus on business, data, and deployment aspects, as well as the modeling, which used to be the main focus. Today, data science practices are more mature and well tested. Even though different methodologies may use different names for each step of the process, in general, I can logically divide any data science exercise into four phases (See Figure below):

Business Problem Understanding/Use,  

Data Understanding/Use and Preparation, 

Analytics and Assessment, 

Implementation (Deployment and Monitoring).

In the context of big data, these logical phases stay the same; however, some low-level details of data preparation, analysis, and implementation may be impacted.

 We all love acronyms and I have been using DS-BuDAI to refer to this process to communicate with business sponsors and users. The lowercase 'u' represents "Understanding/Use" to overemphasize their importance during Business and Data focused phases. It bridges the two. Analytics and Implementation are simply realizations of the data science deliverable.

The "Understanding" part needs no explanation specially in the context of business problem and data that are specifically going to be addressed and leveraged in the effort. "Use" however needs a bit of explanation given some recent experiences. 

A DS project must start with a full understanding of the business challenge and how it could be solved leveraging data sources available or to be obtained. However, there could be cases that after everything is done and the value proven, the business users are not still willing to use the new insights for actions. This lack of responsiveness has a lot to do with the culture of the organization, how decisions have been historically made in the past, and the marginal improvement the new actions will bring. These however could be overcome with education and training and full support of senior management for change.

In some cases though, actionable insights are perceived by business users as "this is what we already knew" and "it is good that the data analysis confirms that." Basically saying that there is no novel new findings but a confirmation of what is known. There is truth to this perception sometimes but at times it is simply resisting change or accepting changes in practice. 

   

In the context of data, "Use" also is essential. Collection, storage, preparation, and management of big data is still expensive no matter how much the storage costs have dropped in recent years with advent of open source systems and price drops in storage/processing systems. Data could easily be abused or misused. Sometimes too much data is used, and sometimes data is not used at the right level of details or aggregation. 

The lowercase "u" in DS-BuDAI is to overemphasize understanding and use during business and data focused phases.


Originally published on10/19/16, 11:47 AM Pacific Standard Time

---

[1] "Data Science" is nothing new except the term itself and the level of recent interest for it.

---------------------------------------------------------------------------
I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).