Data

20 posts
4 Ways AI Is Impacting eDiscovery & Review In 2024 & Beyond

4 Ways AI Is Impacting eDiscovery & Review In 2024 & Beyond

So many things changed in 2001. Certainly, there were major political and security changes that came because of the terrorist attack on 9/11. Profound and long-lasting changes.

So, too, there were massive changes in how we conducted discovery in litigation and investigations. Specifically, discovery started to have an “e” in front of it.

Adding the “e” to Discovery 


I remember my first eDiscovery project in 2001 where discussions were held over how to collect electronic information from a computer and whether and how we should print that information to make it ‘usable.’

Slowly, everyone in litigation started to grapple with the “e” but were doing so inconsistently from law firm to law firm. Finally, in December 2006, the federal rules were amended to make their first attempt at grappling with the “e” by defining and standardizing various processes. What was readily accessible? What was proportional? How far back would you have to look and how broad would the searches be?

We had asked similar questions before but more as it related to file cabinets and filing rooms in storage facilities. Now we were grappling with backup tapes and servers and floppy disks.

The Introduction of Machine Learning 


I have long said that you deal with technology problems by using technology. When the volume of data became humanly unmanageable, we started to see the litigation world go through new changes as it relates to the use of technology. Machine learning was introduced on various review platforms. Humans made decisions that computers would then enhance and find similar documents.

The acceptance of such machine learning has not been universal. In the age of ever-changing technology, the legal landscape never seems to keep up with the business realities. Mobile devices, remote work, and all its implications are dealt with daily in the business world and in litigation. However, lawyers both by training and experience do not always have the knowledge base or the skill sets to answer the questions that inevitably arise. We need to close that knowledge gap.

AI Is Taking Over 


​Now, in 2024 we start to grapple not with the “e” in discovery but with artificial intelligence (AI) and its impact on the eDiscovery process. We will look at four areas that will inevitably be impacted by the use of AI and other machine-learning tools. First, we will look at the impact on the collection of data. Next, we will look at the impact on searching that data, particularly as it relates to protected content. Third, we will look at the growing impact on the review process. And finally, the impact of AI on the analysis of the data that we need for deposition or trial (i.e., the use of the data).

Collecting Data


No longer can you collect everything. There’s just too much. Data may be cheap to store, but it’s not cheap to collect, search, and review. We need to find a better way so the cost doesn’t overwhelm the value of the case.

Also, the sources of data are increasing in number. Beyond shared drives and hard drives. We now have multiple social media platforms and websites (mostly containing video instead of just text data.) This makes the use of AI critical to collect only the needed data. And to improve the process from one litigation to the next. AI can collect, preserve, and prepare the data for searching.

Searching 


AI has an incredible searching capability. Not just using antiquated keywords which have been used for more than a decade, but nuanced searching particularly for protected content. Things like attorney-client privilege, work product, or, in regulated industries, additional privileges related to banking and suspicious activity reports. Or, in the healthcare industry, anything related to protected health information, personal identifying information, HIPAA, or any of a number of other privileges that might apply to your particular industry.

We are fast approaching the ability to type or speak out a search using AI that searches, in mere seconds, across the universe of data types that you’ve collected. Then, changing or editing that search and re-doing it equally as fast. Each time, you get a better sense of the data available rather than searching for keywords or having attorneys review each set to see what’s in the data.

Review Is Changing 


When I started as a baby lawyer looking at documents, we had pieces of paper and we had a file room where that paper was stored. We had individuals looking at each page wondering what we might discover as we went through someone’s file. Gradually, we started to use the computer for tagging and consistency between different productions in different locales.

The industry finally moved on to machine learning where individual attorneys could make decisions that the computer would then try to replicate and enhance rationally to include similar documents. (Not unlike how streaming services try to find you movies similar to the ones that you have liked or watched before.)

The growth of AI in the review space is not over. Instead of just assisting the lawyer on that first-level review to find responsive, confidential, or privileged materials, I see AI quickly making that step disappear completely.

Instead, a higher-level, experienced review attorney who specializes in quality control review or in that specific industry will focus on training the AI and then sifting through what the technology has identified as responsive, confidential, or privileged. That quality step is necessary but it will also likely be limited as AI becomes more sophisticated and nuanced and lawyers become more comfortable with its outputs.

Analysis


​This would include documents you are producing for the other side as well as documents you are receiving from the other side.

AI can go through an enormous amount of information in a short amount of time. It can categorize it and put documents into subject matter buckets considerably faster than a human can. It can also spit out its analysis of what it has categorized.

AI no doubt will be increasingly used for this step possibly removing the need for the young associate at the outside counsel and, much like in the review world where only quality reviewers are remaining, only those dealing with the actual litigation will need to look at those documents.

Embrace It


Technology has impacted the legal field for at least the last 20 years. The speed of its impact is only going to increase. Both in terms of jobs available and the in-depth analysis that is available now, that would have taken a human many, many hours to absorb and understand.

I am of the persuasion that we should not fight technology advances. While the ATM did cause a shift in the banking employment industry, I much prefer going to an ATM to get cash than standing in line in front of a teller. Restaurants, too, were impacted by the pandemic and the push towards drive-thru and delivery versus customers dining inside. They had to adjust their business model. So, too, will lawyers.

We need to embrace it, we need to understand it, and we need to learn to use it rather than creating roadblocks to try to preserve what has gone before.

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

4 Ways AI Is Impacting eDiscovery & …

5 Trends In Market Research To Watch Out For In 2024



For the past 20 years, if not longer, the importance of businesses keeping up with changing consumer expectations and increasing demand for better experiences was already well known to be crucial to market success. After all, that was the basis for the existence of market research: to provide organizations with know-how into consumers’ behaviors, needs, and wants to improve their marketing, advertising, products, and services.

Technological advancements have contributed to the changes in consumers’ needs, wants, and expectations; consumers are more informed now, more connected to the brands they use and what other consumers are saying, expect better and faster experiences, and are thus more demanding of the brands they interact with.

The rate of development of these paradigm shifts among consumers has only gained speed since the global pandemic and now, more so than ever before, businesses need to understand, and even attempt to anticipate, these changes before they occur if they want to fuel innovation and transform their business models, operations, marketing, and the way they develop new products, services, and experiences if they want to stay relevant in the market.

As with business, the market research industry has experienced great change, and it is now almost impossible to determine the root cause of any emerging trend as a people issue or as a byproduct of technological advancements, but rather a combination of both.

Within the emerging trends observed within the industry, five tend to stand out over others for their transformational impact beyond the industry itself.

1. Artificial Intelligence (AI) And Machine Learning (ML)


Artificial intelligence (AI), machine learning (ML) concept

The further incorporation of artificial intelligence (AI) and machine learning (ML) within the industry will bring forth not only the ability for organizations to analyze new sources of data, evaluate more data faster, and identify insights around current consumers we could not obtain before, the combination of access to more data, artificial intelligence algorithms and machine learning will enable organizations to anticipate market shifts through the adoption of predictive and prescriptive analytics, a sub-trend that will continue to develop in the upcoming year.

While its advantages and disadvantages with respect to research are still being discussed, the use of synthetic data—data generated to imitate information from real consumers—will also be explored further to circumvent issues regarding the availability of data, eroding consumer data quality, and privacy concerns for participant information.

2. Integration With CRM platforms


CRM concept

As the dynamism with which multiple sources of data are evaluated to gain deeper insights, CRM platforms are the perfect conduit to provide information about consumers that could not be effectively obtained and analyzed before (much less in real time!). While their full integration is still a challenge in many organizations, the advancements made to date are beginning to reap fruit and the analysis of internal consumer data with market research data and other sources will continue to grow moving forward.

3. The Rise Of In-House Qualitative Research


Qualitative research concept

With the increased reliability on CRMs, the personalized experiences these could provide consumers, and the scale of qualitative work that could be achieved through them, businesses will be looking to conduct more qualitative work internally. This is where the idea of bringing experienced qualitative researchers in house will be of great value and organizations that do not will risk not leveraging the full potential of CRM platforms.

4. From Servicing To Becoming Advisors, To Collaborators


Business people collaborate on market research project

With the occurrence of the above-mentioned trends, the role of the market research supplier will evolve accordingly. With the further growth of CRMs within organizations and as these become more adept at looking for insights internally, it is clear the role of the market research supplier will need to continue to evolve as well.

The role of finding the insights will no longer be left to the research entity but rather will complement internal teams on leveraging their internal CRM platforms and other sources of information to establish as close a 360° view of the consumer as possible, build strategic plans for success, and identify how to measure that success moving forward.

5. Ethics & Privacy


Data privacy, ethics concept

Data privacy and corporate transparency around how consumer data is gathered and used will continue to play a critical role in the industry as it continues to evolve. Organizations will look to follow GDPR and CCAA policies more closely and adopt transparency-centric ways of data collection that can be personalized by consumers.

The year 2024 will be one of continued transformation for the market research industry. New technologies, innovative methodologies, and new ways of collaboration between research agencies and their clients will continue to share center stage for those organizations that wish to move ahead in the market, while those that do not do so will risk their permanence in the market.

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

5 Trends In Market Research To Watch …



It’s a normal workday, and then suddenly the fire alarm goes off in the back of the server room. Amidst the chaos, do the IT staff know what they’re supposed to do besides call 911 and make sure everyone is safe? A comprehensive IT disaster recovery (DR) plan is your playbook. It defines the IT infrastructure and how to restore and resume critical IT functions minimizing the impact on the business including downtime, data loss, and reputational damage.

Start by performing a risk assessment identifying key threats and vulnerabilities to your IT infrastructure including hardware/software failure, cyberattacks, natural disasters, and major power outages. What is the likelihood of each incident and the subsequent impact (enterprise-wide, regional, department-specific, etc.) for your organization? Now you can prioritize your recovery efforts accordingly.

Next, create a comprehensive DR plan that includes the who, what, where, when, why, and how details. For example:

  • Who = What are the main roles and responsibilities including identifying any essential IT staff? For example, did you have any IT staff turnover? You’ll need to notify stakeholders such as management, vendors, and suppliers so you need the correct contact information.
  • What = Which systems are mission critical, essential, and non-essential? Criticality will define the order of recovery and ensure that the most critical systems are restored first.
  • Where = Do you have global data centers or IaaS? A hot site or an alternate cloud-based infrastructure for temporary operations?
  • How = Step-by-step procedures for recovering systems, identifying which processes need to be done in a specific order.

Backup And Recovery Strategy


Data recovery concept

​You need a solid backup and recovery strategy. Perform regular backups including software, configurations, and data. Backups can be onsite, offsite, cloud-based, or a combination depending on the business needs. For mission-critical systems, it may be beneficial to implement redundancy (e.g., deploying backup systems, or redundant hardware) to minimize the impact of an incident.

The IT environment and the business are continually changing so the DR plan needs to be kept current too. Make sure you regularly review, update, and refine the plan to make sure it reflects any changes in your environment. Did you implement a new ERP system or relocate your data center? Or did you have any technology changes that improve the recovery process or enhance the resilience of your systems? If so, you should take the time to update your DR plan.

Testing Your DR Plan


Business meeting to talk about disaster recovery plan

Make sure you regularly test the DR plan which will validate recovery procedures, identify gaps, and ensure data such as stakeholder contact information is correct. There are multiple ways to test including tabletop and scenario-based simulation. Tabletops are good, but simulations including recovery are better. It’s key to test on a regular basis...with the business (that takes your plan to the next level)! The frequency of your testing may vary depending on the criticality of the systems or any regulatory requirements so stay up to date with any regulation changes that may impact your DR plan requirements. Make sure you document lessons learned (including from any actual incidents) so that you can identify any areas for improvement.

It’s critical to ensure your DR plan is aligned with the business continuity plan. Within the DR plan, there should be a communication component. When an incident occurs, you need protocols to notify and continually update the stakeholders so that they can respond appropriately.

Encourage a culture of continuous improvement. Encourage and solicit feedback for the DR plan. Although IT is responsible for disaster recovery, it affects the entire business continuity plan. Have a comprehensive and current DR plan that aligns with the business needs, and which is tested regularly. It enhances your readiness for the next disaster and minimizes the impact of disruptions to your IT infrastructure as well as minimizing the impact of potential disruptions.

For more information on taking your DR plan from good to great, follow me on LinkedIn!


Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

Is Your IT Disaster Recovery Plan Good …

Actionable Reporting (And AI) In A Data-Driven World



Organizations have a plethora of data and need to collect and transform it into information and actionable reporting. The business wants relevant, accurate, and timely information for decision-making, problem-solving, and continuous improvements.

For example, information may show trends or identify issues that need improvement or attention to improve performance. And when there is a continuous feedback mechanism, the information can be used to measure the effectiveness of improvement efforts and make data-driven adjustments as needed to achieve better outcomes.

Some organizations are taking it one step further and using artificial intelligence (AI) such as ChatGPT and Bard for additional insights. Organizations have been using chatbots for customer service inquiries and are automating tasks and generating various types of content saving time. Organizations are also using data to analyze performance metrics, identify areas of inefficiency, and even analyze historical data to make predictions about future trends.

AI models can utilize historical data to make predictions, providing valuable insights. Make sure you have a corporate governance policy for AI for responsible and ethical use and minimizing risks. This includes items such as use cases of what it can (and can’t) be used for, where (public AI v. private instance), data confidentiality, etc.

Data Quality


AI, artificial intelligence, data concept

As a result, data quality has become more important than ever. Making sure your data is as clean as possible is a critical step! Some signs you have dirty data are:

  • Data entry errors – individuals sometimes make errors such as misspellings, transposed digits, or other inconsistent formatting;
  • Missing data;
  • Duplicate data; and
  • Data source discrepancies – data from different sources that have inconsistent or conflicting data.

For AI, if your data is inaccurate, incomplete, or contains errors, the output may be misleading. Good data quality contributes to the model's ability to handle various inputs and scenarios effectively. Also, ensuring that your data is diverse and free from biases is essential to creating AI solutions that are fair and inclusive. Otherwise, you may introduce bias resulting in unfair or unintended results.

How do you know you may have a problem? If you get comments from end users that the data seems incomplete or outdated (lagging), you should investigate. Or if you get complaints from external customers about their account information. Collaborate with the data owners or subject matter experts (SMEs) to help identify discrepancies/anomalies and how to correct the data both present and ongoing.

Also, if your organization is the victim of a security breach or unauthorized access, make sure the data hasn’t been changed, corrupted, or contaminated. Take the time to ensure the data is still accurate and reliable.

Data Governance Framework


Data governance

It starts by having a comprehensive data governance framework and should be an ongoing process because data quality is not “one and done.” This includes, but is not limited to:

  1. Data governance framework – have policies and procedures to establish and enforce data quality standards and data ownership within the organization;
  2. Data security – the data owner should determine who should have access to specific data fields. For example, only a small handful of people should be able to access salary/payroll information;
  3. Standardize data collection – create a process to minimize data errors and inconsistencies;
  4. Data validation – validate data being entered to prevent incomplete or inaccurate data from being entered into the system. For example, making key fields required, having valid values and date formats;
  5. Data cleaning – identify and correct any errors such as missing values, outliers, or duplicate records; and
  6. Data quality metrics – continually monitor and report on the quality of the data identifying any areas that need improvement.

Otherwise, you may be a victim of the expression “garbage in, garbage out” which will affect your reporting. You want to make sure your information is relevant, accurate, and timely so that the business has actionable reporting that is reliable and can be trusted.

For more information on the importance of good quality data for actionable reporting and AI, follow me on LinkedIn!

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

Actionable Reporting (And AI) In A Data-Driven …

Executive Spotlight: How Artificial Intelligence (AI) Will Transform Management



As organizations navigate an era characterized by rapid technological advancements, artificial intelligence (AI) emerges as a transformative force that promises to revolutionize the way management is conceptualized, executed, and optimized. While the benefits of integrating AI into management are numerous, from automating routine tasks to enabling data-driven decision-making of unprecedented accuracy, there are also various disadvantages and potential challenges that leaders need to be aware of during this transformative journey.

We recently asked our leading executives how they think AI will transform management.

Here are their responses...

Ana Smith, Leadership Development & Learning Strategist


I can confidently say that AI is poised to transform management in numerous ways, revolutionizing how businesses operate and leaders make decisions. Here are some key ways AI will transform management:

  1. Data-Driven Decision Making: AI enables managers to make data-driven decisions by analyzing vast amounts of data quickly and accurately. AI-powered analytics can identify patterns, trends, and insights that human managers might miss, leading to more informed and effective decision-making processes.
  2. Automating Repetitive Tasks: AI can handle repetitive and mundane tasks, freeing up managers' time to focus on more strategic and creative aspects of their roles. This automation can lead to increased efficiency and productivity across the organization.
  3. Predictive Analytics: AI can predict future outcomes based on historical data, helping managers anticipate challenges and opportunities. This capability empowers leaders to be proactive in addressing potential issues and optimizing business strategies.
  4. Enhanced Customer Insights: AI can analyze customer data to provide deep insights into consumer behavior and preferences. This information enables managers to tailor products and services to better meet customer needs and drive customer satisfaction.
  5. Personalized Leadership Coaching: AI-powered coaching tools can provide personalized guidance to managers, helping them enhance their leadership skills and management capabilities. These tools can identify areas for improvement and offer actionable recommendations.
  6. Talent Management and HR: AI can streamline the talent acquisition process by screening resumes, conducting initial interviews, and matching candidates with suitable positions. Additionally, AI can analyze employee data to identify patterns that can inform HR decisions related to employee engagement, retention, and performance management.
  7. Virtual Assistants for Managers: AI-powered virtual assistants can help managers with scheduling, reminders, and organizing their daily tasks. These virtual assistants can provide valuable support to busy managers, keeping them organized and on track.
  8. Supply Chain Optimization: AI can optimize supply chain management by analyzing demand patterns, inventory levels, and logistics data. This can lead to improved inventory management, reduced costs, and better overall supply chain performance.
  9. Risk Management: AI can help identify and mitigate potential risks by continuously monitoring data and identifying anomalies. This proactive approach can enhance risk assessment and response strategies.
  10. Real-Time Insights: AI can provide real-time insights into various aspects of the business, allowing managers to monitor key performance indicators (KPIs) and react swiftly to changing market conditions.

It is crucial for managers and organizations to embrace AI and adapt to these transformations. Proper implementation and understanding of AI's capabilities can lead to improved decision-making, increased efficiency, and competitive advantage in the ever-evolving business landscape.

Ana Smith helps people & organizations achieve their full talent potential by developing and co-creating people strategies and customized solutions, and turning them into impactful outcomes and collaborative relationships, using coaching as the "red thread."

Michael Willis, Sports Business Operations Executive


Artificial intelligence (AI) in football

I look at artificial intelligence (AI) as historical, technological breakthroughs like flying, the automobile, electric lights, and the telephone due to their transformative impact on society.

In comparing AI to these historical discoveries, it’s important to note that while these innovations brought about significant changes, AI’s impact is particularly noteworthy due to its potential to affect nearly every aspect of human life, from work and education to healthcare and entertainment.

Additionally, AI presents unique challenges and considerations, such as ethical concerns, potential job displacement, algorithm bias, and questions about how AI will integrate into our social fabric. As AI advances, its trajectory will likely be shaped by technological progress, societal adoption, regulatory frameworks, and ethical guidelines.

Artificial intelligence (AI) is already impacting various aspects of sports management, and its transformative effects are expected to continue growing. Here are some ways AI could transform management in sports:

Player Activities

  • Injury Prevention and Management
  • Nutrition and Conditioning
  • Scouting and Recruitment
  • Game Strategy and Analysis
  • Data-Driven Management Decisions
  • Sports Analytics and Insights

Game Day Activities

  • Fan Engagement
  • Ticket Pricing and Sales
  • Social Media and Branding
  • Referee and Umpire Decisions

Striking the right balance between technology and human expertise will be crucial to successfully harnessing AI’s benefits in the sports industry.

Michael Willis has 18+ years of experience working with accounting & sports organizations and has managed P&Ls of $10M - $125M+ with budgets of $3M-$50M+. He worked for the NFL for 22 1/2 years, mainly with the game officials working on the financial/accounting side of the business.

Lynn Holland, GTM & Rev Gen Consultant


Artificial intelligence (AI) concept

Coming from a perspective of selling operational transformation through technology adoption for efficiency, increased capacity with the same assets, and financial gains, I am a proponent of advancement.

As such, I see impactful advantages for managers to get behind an initiative to infuse AI tools into their management process. At the same time, I caution executives of the potential downside and to anticipate several obstacles when seeking buy-in with their teams:

The Managerial Advantages of AI:

1. Balancing the skill sets, workloads, and availability of subordinates to ensure allocated tasks and resources maximize productivity.

2. Monitoring employee performance metrics in real time to provide immediate training and coaching rather than waiting for formal reviews.

3. Gaining actionable insights as AI spots historical trends and patterns and predicts future trends and outcomes to make better decisions, pivot quickly, and substantiate management decisions for subordinates.

The Downsides & Obstacles:

1. Potential loss in empathetic support, understanding, and personal interaction between managers and their employees.

2. Erosion of managerial critical thinking and decision making without an AI tool.

3. Creating a vibe for employees that they are being surveilled by “Big Brother” with diminished autonomy, protections for personal information, and a sense of personal space.

4. Employee dependence upon turn-by-turn instructions, eroding the same critical thinking and decision making.

5. Employee fear of displacement or a skills gap to incorporate AI tools into their workflow.

Lynn Holland is a business development executive with 18+ years of experience taking operational, IoT & retail technologies, products, & consumer engagement to market with a focus in petroleum & convenience retail.

Lisa Perry, Global Marketing Executive


Artificial intelligence (AI) concept

One of the most significant disruptors in this era of rapid change is artificial intelligence (AI). Nowadays, 77% of businesses are using or exploring AI. Among them, 35% of companies are using AI, and 42% are exploring AI for its implementation in the future. AI is reshaping the landscape of organizational leadership and management in several ways.

  • Data-Driven Decision-Making: AI can process vast amounts of data in real-time, providing managers unparalleled insights into customer behaviors, preferences, and sentiments leading to higher engagement and strong brand loyalty.
  • Predictive Analytics: Leveraging predictive analytics, managers can anticipate market trends, customer preferences, and potential risks enabling organizations to adapt to changing circumstances and giving them a competitive advantage.
  • Enhanced Efficiency: Using AI-powered tools allows management to focus on high-level strategic thinking and fostering innovation.
  • Strategic Planning: AI can assist management in formulating well-informed strategic plans by analyzing market trends and competitive landscapes much faster rate.
  • Customer Insights: AI-powered analytics can process customer data from various sources, providing comprehensive insights into their preferences, behaviors, and sentiments. Management can tailor products, services, and marketing campaigns to meet customer expectations effectively.

AI is the most transformative technology we will see in this lifetime. We better integrate it into our skill set, leadership, and management practices quickly to add value and ROI to the bottom line.

For a step-by-step guide on how to develop a brand strategy, check out my book How to Develop a Brand Strategy.

Lisa Perry helps companies build leadership brands, driving loyal customers & delivering profitability. She does this through a process that builds brands consumers love. Her goal is to help companies develop, monetize, and grow their brands.

Duncan Prior, Digital Transformation Consultant


Artificial intelligence (AI) concept

Artificial intelligence has arrived and is here to stay and there are three areas where AI will transform management:

1. Managing the current emergence of AI

Robert Glazer devoted a "Friday Forward" weekly article to discussing the emergence of AI in the context of all other technology innovations and I reflect on parallels with the dot-com era. The article breaks down the steps in the innovation wave that engulfs us until things finally settle down and organizations can refocus on their objectives. For now, managers can expect the subject of AI to surface in most conversations so they will have to have a plan for dealing with the current hype.

2. Managing legacy

Perhaps the biggest challenge any organization faces is the delivery of digital transformation and change. From a technology perspective, the key reason is the legacy IT infrastructure that most companies continue to run. Cloud platforms and mobile devices offer the promise of supporting an agile business; however, getting there is rarely straightforward. Without that transformation, AI will not deliver any benefits and, if introduced on top of an aging business platform, will only introduce more complexity and chaos.

However, AI will deliver on that promise either for new business models or for organizations that have been able to transform, so the pressure on those managing a legacy business will increase over the coming years.

3. The day-to-day possibilities

For those running flexible IT systems, help is at hand that will enable managers to free up time to transform their organization. AI tools provide the opportunity to act as a "co-pilot" for staff. Take the example of submitting proposals for new business. This can be a time-consuming exercise with often a small chance of success. Most of the time is spent assembling the proposal response which takes up too much of the team’s time when it could be focussed on the true value of its offering. Using AI to assist with this work isn’t cheating. Rather, it is helping humans to perform their jobs as effectively as possible. And, in this example, AI can help people focus on opportunities that they have the most chance of winning!

Duncan Prior is a consultant specializing in digital transformation delivery and the host of the Making Things Work podcast series. He enjoys running, reading, and listening to music.

How do you think AI will transform management? Join the conversation inside Work It Daily's Executive Program.

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

Executive Spotlight: How Artificial Intelligence (AI) Will …

Executive Spotlight: What AI Means For The Future Of Work



Artificial intelligence (AI) is intelligence (such as learning, reasoning, and inferring information) demonstrated by computers that mimics human intelligence. In the workplace, it can be used to perform cognitive abilities and tasks with reduced human error and cost. But this is just the tip of the iceberg of the transformative potential of artificial intelligence for the future of work.

We recently asked our leading executives what they think AI means for the future of work.

Here are their responses...

Ana Smith, Leadership Development & Learning Strategist


In continuation of our exploration of artificial intelligence (AI) and its impact on the future of work, this aims to shed light on the potential drawbacks and challenges associated with the integration of AI in the workplace. While AI offers tremendous opportunities, it is essential to acknowledge and address the concerns that arise alongside its implementation. Let us delve into the cons of AI and foster a comprehensive understanding of its implications.

  1. Job Displacement and Transition: As AI technology advances, there is a legitimate concern regarding job displacement. Tasks and roles that can be automated may lead to workforce reduction or shifts in job requirements. Professionals in industries heavily affected by automation may need to navigate transitions and acquire new skills to adapt to changing job market demands. It is vital to implement effective reskilling and upskilling programs to support individuals in successfully transitioning to new roles.
  2. Ethical Considerations and Bias: AI systems rely on large amounts of data, and if this data carries inherent biases or reflects societal inequalities, it can perpetuate discriminatory outcomes. Algorithmic bias can impact hiring practices, decision-making processes, and access to opportunities. Ethical considerations surrounding AI, such as privacy concerns and data security, must be addressed to ensure fairness, transparency, and accountability in the workplace.
  3. Overreliance on AI and Lack of Human Judgment: While AI excels at processing vast amounts of data and making predictions, it may lack the ability to understand complex contexts and exercise human judgment. Overreliance on AI systems without human oversight can lead to errors, misinterpretations, or flawed decision-making. Maintaining a balance between AI automation and human expertise is crucial to mitigate risks and ensure the responsible use of AI technologies.
  4. Workforce Adaptability and Learning Curves: Implementing AI technologies often requires a learning curve and adaptation for the existing workforce. Not all employees may be technologically adept or comfortable with AI integration. Organizations must invest in comprehensive training programs and provide ongoing support to help employees navigate the changes effectively. Overcoming resistance to change and fostering a culture of continuous learning are essential for successful AI implementation.
  5. Job Satisfaction and Work-Life Balance: AI's ability to automate certain tasks may inadvertently increase workloads and expectations for employees. As AI takes over repetitive and mundane tasks, professionals may find themselves handling more complex responsibilities, potentially leading to increased stress levels and a compromise in work-life balance. Organizations must proactively address these concerns to ensure that AI implementation enhances employee well-being and job satisfaction.

While artificial intelligence presents remarkable opportunities for the future of work, it is crucial to recognize and address the associated challenges. By acknowledging the potential drawbacks of AI, such as job displacement, bias, ethical considerations, and workforce adaptability, organizations can proactively navigate these concerns. Striking a balance between AI integration and human judgment is key to harnessing AI's potential while safeguarding the well-being and success of the workforce.

Ana Smith helps people & organizations achieve their full talent potential by developing and co-creating people strategies and customized solutions, and turning them into impactful outcomes and collaborative relationships, using coaching as the "red thread.

Michael Willis, Sports Business Operations Executive


Man uses artificial intelligence (AI) technology at work

I want to look at AI from the standpoint of leadership selling the concept to C-suite executives.

Selling Up:

1. Understanding C-Suites Destination

These executives are visionaries who look at things in the time and space of decades into the future. Their foundation is mission, vision, and purpose.

It would be best to build a narrative; be a good storyteller, a master at telling their story.

You must show them where they are now and where they want to go. The progress of everything in the middle will get their attention.

If Henry Ford had asked the public if he could help them with something to improve their lives, the day’s response might have been we’re good; all I need is a faster horse.

2. Articulate the Value

You might see the value in AI and all that it can bring to the sports world. But your task is to articulate what’s important to them and concerned about every day.

Tell them something that they are not aware of.

You need to understand the observable group behavior of the culture of the C-suites. The essence of a person’s culture is a group of behaviors that you can measure, see, and experience. So, the C-suite will have a culture associated with it.

And there are individual cultures that influence their behaviors and decisions. Still, then there’s the group culture that affects the group and, therefore, the individual’s decisions to have a unanimous decision to buy into what you’re recommending.

So that’s why it’s essential to understand how and what areas their culture influences.

3. Trust

What is the C-suite modus operandi concerning trusting? For them to trust your pitch on AI, they must trust it and you.

Once you have articulated the three steps in the order that I have laid out from their perspective, from their lens, you’re on your way.

Finally, you must evoke the right emotions. Emotions will have the final say. It’s not the idea they’re buying into—not the thing itself. It’s the meaning behind it that’s important to them.

Michael Willis has 18+ years of experience working with accounting & sports organizations and has managed P&Ls of $10M - $125M+ with budgets of $3M-$50M+. He worked for the NFL for 22 1/2 years, mainly with the game officials working on the financial/accounting side of the business.

Lisa Perry, Global Marketing Executive


Artificial intelligence (AI) technology concept

As a brand marketer, I cannot ignore the profound impact that rapid advancements in artificial intelligence (AI) have had on various industries, and my industry is certainly no exception. AI technology is revolutionizing how companies engage with their audiences, streamlining processes and making data-driven decisions. Here are four ways AI is changing the future of work in brand marketing and how businesses can leverage this transformative technology to gain a competitive edge:

Automating Routine & Repetitive Tasks

AI technology offers the potential to automate routine and repetitive tasks, freeing up valuable time for brand marketers to focus on strategic initiatives. Chatbots, for example, can handle customer inquiries, providing instant responses, personalized recommendations, and support around the clock, enhancing customer satisfaction and allowing marketers to allocate their time and energy towards creative endeavors and relationship-building efforts. Additionally, AI-powered tools can automate social media posting, content generation, and campaign optimization, increasing efficiency and productivity.

Enhanced Data Analytics & Consumer Insights

One of the most significant advantages of AI for brand marketers is its ability to process vast amounts of data and unlock valuable consumer insights. AI-powered analytics tools can analyze consumer behavior, preferences, and patterns, enabling marketers to create more targeted and personalized campaigns. By leveraging AI algorithms, marketers can segment their audience more effectively, tailor content, and optimize marketing strategies based on real-time feedback, resulting in higher engagement and conversion rates.

Predictive Analytics & Forecasting

AI’s analytical capabilities enable brand marketers to predict future trends and make data-driven decisions more accurately. By leveraging machine learning algorithms, marketers can analyze historical data to forecast market demands and consumer behavior and identify emerging opportunities. This empowers them to anticipate shifts in the market, refine marketing strategies, and optimize resource allocation. With AI-driven predictive analytics, brand marketers can stay ahead of the competition and adapt proactively to changing market dynamics.

Hyper-Personalization

AI empowers brand marketers to deliver hyper-personalized experiences to consumers. By leveraging AI algorithms and machine learning, marketers can analyze user data to understand individual preferences, purchase history, and behavior patterns. With this knowledge, brands can create highly targeted marketing campaigns, personalized product recommendations, and tailored messaging that resonates with each customer on a deeper level. This level of personalization fosters stronger customer relationships, boosts customer loyalty, and ultimately drives revenue growth.

Artificial intelligence is transforming the future of work in brand marketing, offering many opportunities to increase efficiency, improve customer engagement, and drive business growth.

Lisa Perry helps companies build leadership brands, driving loyal customers & delivering profitability. She does this through a process that builds brands consumers love. Her goal is to help companies develop, monetize, and grow their brands.

What do you think AI means for the future of work? Join the conversation inside Work It Daily's Executive Program.

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

Executive Spotlight: What AI Means For The …



In today's world, digital technologies have transformed how brands operate and interact with customers. With the rise of social media, mobile devices, and other digital channels, it has become imperative for brands to have a strong online presence and engage with their customers meaningfully. Brands that navigate the digital landscape successfully can reach a wider audience and build stronger customer relationships.

In this context, brands must understand the key factors contributing to digital success. Here are five ways brands can succeed in the digital world.

Traditional Marketing vs. Digital Marketing


Brands need to be aware of some key differences between traditional and digital marketing strategies to succeed in the digital world. The first thing to recognize is it's no longer a traditional vs. digital marketing world. It's all about being customer first and knowing how they (in their segment) journey through your brand, regardless of what touchpoint they enter. It's about multi-touchpoint marketing and how you drive them across the marketing funnel—from brand awareness to consideration to conversion to loyalty and advocacy.

It's no longer a linear journey. Consumers may come in at any touchpoint. It also realizes that brands are playing the role of conductor. How are you measuring the effectiveness of each channel and then optimizing your results to achieve your goal? Finally, it's a balance between a brand's owned, earned, and paid media.

Personalization


Woman on phone receives personalized content from brands

Personalization is important in today’s digital landscape, and there are some effective ways to tailor marketing messages and content to individual customers. The first thing is to break out your data into customer segments and understand their needs (and the benefits your brand can bring them) and then how they consume media / what channels they are on and everything about them so that you can present your brand as a solution to fit their needs. Then create personalized messaging, content, and offers per segment.

For example, in broad strokes, older audiences will be on Facebook and reading emails, while a younger demographic will typically be on TikTok and Instagram, viewing YouTube Shorts, and getting their marketing messages via opting into SMS text messaging.

Build Trust & Credibility


Young people on their phones laugh at a video from one of their favorite brands

In the digital world, building trust and credibility is essential for brands to succeed. However, it can also be one of the biggest challenges they face. One of the main obstacles is the abundance of fake news, misinformation, and online scams that can damage a brand's reputation. Another challenge is the lack of face-to-face interactions, which can make it harder for customers to trust a brand. Additionally, there is the issue of data privacy, with many customers being concerned about how their personal information is being used.

To overcome these obstacles, brands need to be transparent and honest in their communications and actions, provide quality products or services, and prioritize customer satisfaction. They can also leverage social proof, such as user reviews and testimonials, to build credibility and trust. Moreover, brands should establish a strong online presence and engage with customers regularly through social media and other channels to foster a deeper connection and build a community of loyal customers.

Social Listening


"We listen to our customer" concept (marketing, branding)

Social listening has become a critical component of any brand’s digital strategy. It is important for brands and is often overlooked, even though there are a lot of lower-cost social media management tools that now offer social listening for a fee. Social listening can help brands see what social conversation is happening (and trending) about their brand globally and nationwide. Consistent social conversation volume can help drive brand awareness. Social listening can also help you discover influencers talking about your brand, and you can reach out to them for influencer campaigns/partnerships.

A second way you can use social listening is for customer service. You can see customers discussing issues about a product or if they’re unhappy with a brand. You can respond with corporate communications as needed to customers or use the information to inform your product development and marketing teams. You can also use it for crisis management. I remember being at a Fortune 500 brand, and a licensee had changed the interface of a popular game. Social listening had picked up the noise of angry fans coming to the corporate office to protest. Because we picked this up with our social listening tool, we triggered a corporate crisis plan immediately, and it defused the situation.

Lastly, you can use social listening for real-time analytics at conferences or events. For example, I ran the social media war room for a major gaming brand at E3 (the Electronic Entertainment Expo). We picked up consumer sentiment on the games and gaming platforms in real time and were able to flex our marketing/PR events and communications in real time to react to our competitors' messaging.

Data & Analytics


Digital marketing / data and analytics concept

Data and analytics have become essential for brands to understand their customers, track their marketing ROI, and optimize their campaigns for maximum impact. Brands can leverage data and analytics in several ways to better understand their customers and improve their marketing efforts.

First, brands can use data analytics to track customer behavior and preferences across multiple channels, such as social media, email, and website interactions. By analyzing this data, brands can identify patterns and trends in customer behavior, preferences, and sentiment toward their brand. This can help brands develop more targeted, personalized marketing campaigns that resonate with their customers.

Second, data analytics can help brands measure the effectiveness of their marketing campaigns and track their return on investment (ROI). By analyzing metrics such as website traffic, conversion rates, and sales, brands can determine the most effective campaigns and make data-driven decisions to optimize their marketing spend.

Finally, data analytics can help brands optimize their campaigns for maximum impact. Using A/B testing and other analytical techniques, brands can test different campaign strategies and optimize their messaging, targeting, and creativity to improve engagement and conversions.

In conclusion, the digital landscape provides vast opportunities for brands to connect with their audiences and achieve their marketing goals. By staying attuned to customer feedback, personalizing messaging, leveraging data and analytics to optimize their campaigns, and prioritizing transparency and authenticity, brands can establish a strong digital presence and drive meaningful results. The digital world is constantly evolving, but by keeping these key strategies in mind, brands can stay ahead of the curve and thrive in the digital age.

For a step-by-step guide on how to develop a brand strategy, check out How to Develop a Brand Strategy by Lisa Perry. For more digital marketing insights, follow Dana Long and Lisa Perry.

This article was written by Lisa Perry and Dana Long.

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

How Brands Can Succeed In The Digital …



Your organization bought an expensive enterprise resource planning (ERP) system, and it came with a comprehensive set of documentation including user manuals, training materials, and technical guides. You utilized the documentation and successfully implemented the system, but how much have you used the documentation since then?

Technology is an investment. You want to maximize the use of the system, including that the system’s operations are consistent and reliable. You regularly update the system for upgrades and patches. It’s crucial to make sure your corresponding documentation is up to date because it enables employees to be knowledgeable about the updated system.

Documentation captures critical information preserving that knowledge and making it available to all appropriate individuals. If only a few specific individuals know how the system works, what happens when they get sick, go on vacation, or resign? The solution is to make sure the information is written down and to document that knowledge.

User-Related Documentation


User documentation supports onboarding and training to ensure employees know how to properly use the system. This allows employees to get familiar with the system so that they can leverage it effectively. User manuals provide end users with guidelines including step-by-step instructions on how to use the system. Security roles identify what the staff can/can’t do versus what requires a supervisor to override. If you’ve added a module, make sure you add the new feature/functionality details so that the documents aren’t incomplete or missing information.

Training materials (e.g., instructor/student manuals, cheat sheets) are used to train both new and existing employees poised for promotions and transfers. Training materials can be used for scheduled training classes or for train-the-trainer scenarios wherein designated trainers (department super users or subject matter experts) train the rest of the organization.

If the documentation isn’t up to date, it can confuse or make it difficult for end users to properly use the system. Allow users to submit feedback (e.g., via a service ticket) regarding documentation such as unclear steps, incorrect screen print, or missing information.

IT-Related Documentation


For the IT staff, documentation serves as a reference for how the system is intended to be used. This includes system setup/configuration, system maintenance, as well as troubleshooting.

Many organizations operate 24/7 so it’s crucial that the IT staff be able to minimize any downtime. Some key documents are:

1. System architecture documents – provide an overview of the system components and interfaces that are interconnected. If you need to migrate your environment from on premises to the cloud, having current architecture documents can minimize the uncertainty of what components you have and how they’re connected.

2. Data-related documents (e.g., data dictionaries and data flow diagrams) – document the data structures. If a table is dropped, do you know which processes including reports will be affected?

3. System maintenance documents – did the upgrade affect the patching process? Does the nightly processing schedule need to be updated? Or any of the backup schedules (daily, monthly, quarterly, etc.)?

4. Troubleshooting documents – guidance on diagnosing and resolving common technical issues. When you upgraded the database version, did you become familiar with the new system messages and codes in advance?

When the system has been changed, make sure the documentation has been updated to reflect those changes. You can use a date stamp in the footer and turn on versioning to identify when the documentation was last updated.

Also, don’t forget to update the disaster recovery and business continuity documents. In the event of a disaster, you want to ensure the organization can minimize any business disruption. It can be chaotic when a disaster occurs, and up-to-date documentation can reduce stress, especially for employees filling in for absent employees.

For more information on the importance of keeping system documentation up to date, follow me on LinkedIn!

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

How Important Is Up-To-Date System Documentation?

Data Fabric Or Data Fabrication?



Earlier in my career, as a data analyst, the biggest area of frustration for my marketing clients was the lag time between their requests and the business insight. The lag was extended if the required data had to be ingested from a new source. The old ETL framework and associated data management tasks created long waits for impactful insights to reach marketing tactics.

Over the course of my career, despite advances in data strategies and infrastructure, similar questions remain:

  1. “Why can’t I get my analytics sooner?”
  2. “Why can’t I access my own data”
  3. “Why does it take so long to source new data that we need for analytics?”

A recent concept that seeks to relieve the stress associated with these questions is the data fabric. According to Gartner, a data fabric is...

“An architecture pattern that informs and automates the design, integration, and deployment of data objects regardless of deployment platforms and architectural approaches.

"It utilizes continuous analytics and AI/ML over all metadata assets to provide actionable insights and recommendations on data management, integration design, and deployment patterns.

"This results in faster, informed, and (in some cases) complete automation of access and sharing.”

Quite a long multi-layer definition, but based on this promise of data fabrics, data analytics teams can now:

  1. Avoid having to move data
  2. Have better access controls and democratize access
  3. Provide faster, more automated analytics through AI

Business managers often think about these issues and their impact on their business:

Speedier time to insight (shorter cycle times): No more ETL, or ELT, or other time-intensive data management tasks associated with data analytics. AI/ML tools will automate data tasks and data virtualization will allow analysts to go directly to the source elements avoiding time-intensive access and exploration—with the exciting possibility of eliminating extract tools, schedulers, and more.

Expense savings: By going directly to the source, no intermediary platforms, systems, or repositories will be needed. Therefore platform/tool expenses and recurrent maintenance expenses would be saved.

Why Move Data (When You Don’t Have To!)


Data fabric, data transmission

The problem many firms face is the storage of their data in many silos and systems—clouds (lakehouses), warehouses, source systems, ODSs, CRM systems, marketing marts, and in some cases even legacy SAS files. Each of these silos introduce workflows that are necessary but increase time-to-insight. How can organizations use all their data efficiently?

The idea behind the data fabric is to break down data silos and get data into the hands of the users. The data fabric is a tapestry connecting data across all platforms to users, creating efficiencies by not having to move data. Data automation capabilities within the fabric further drive efficiencies by providing accessible, quality data for use by AI and other tasks.

One of the key enablers of a data fabric is a virtualization layer that provisions data needs directly (from transaction and operational systems) without moving or copying data. Data tasks like sourcing, extracting, cleaning/transforming, etc. are all automated. The fabric also helps manage the lifecycle of data, for example, by governing your data using active metadata to enforce the policies, including access, compliance, and quality.

In summary, data fabrics are still evolving—the architecture has merit, the concept seems more developed than that of the data mesh (which is much more of a theory), and foundational tech capabilities are well-enabled. I see fabrics as having great promise, but more as an architecture strategy with a focus on less data movement, fewer silos, more access to data, and faster analytics. Will fabrics eliminate the need to move data completely? I don’t think so (at least in the near term), as there will always be a need to move and replicate some data. Would love to hear from the vendors in the space on this post as well.

Is it data fabric or data fabrication? Only further development (and time) will tell, but I’m betting on the fabric’s promise!

I’m also betting on you, my readers—I bet that a discussion with you on this topic will surface many challenges and opportunities for data fabrics. What would you like to learn more about? What has been your experience with the data fabric? Is it fabric or fabrication? Please reply to this post with your comments so we can all dive deeper into the strategic, tactical, and business implications of this fascinating area.

Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

Data Fabric Or Data Fabrication?

The Data Science ROI Of Moving To The Data Lakehouse



Many CDOs/CDAOs (and even CMOs) have raised some interesting questions about these three data management and data science titans: Snowflake, Databricks, and DataRobot. Some of the questions include:

  • Can Snowflake replace the CDP?
  • Do I need Databricks if I have ML Ops capabilities or other capabilities that are similar?
  • What is the difference between DataRobot and Databricks?
  • How do we prevent the siloing of marketing data and smooth access by other domains?

    Later in this post, I’ll be discussing these questions with Ed Lucio, a New Zealand data science expert for Spark (telecom provider) and former lead data scientist for ASB Bank. We’ll be giving our POV on these questions as well as highlighting a few data analytics use cases that can be driven by these tools once they are in place. I would love to hear from you regarding other use cases and your experiences with data lakehouses.

    Before diving into my conversation with Ed, a quick overview of environments and tools...

    Types Of Storage Environments


    Data cloud, data storage concept

    We, as an industry, have gone from the data warehouse to data lakes, and now to data lakehouses. Here’s a brief summary of each.

    The data warehouse: Closed format, good for reporting. Very rigid data models that require moving data, and ETL processes. Most cannot handle unstructured data. Most of these are on-prem and expensive and resource-intensive to run.

    The data lake:

    Pros

    • Handles ALL data, supporting data science and machine learning needs. Can handle data with structure variability.

    Cons

    • Handles ALL data, supporting data science and machine learning needs
    • Difficult to:
      • Append data
      • Modify existing data
      • Stream data
    • Costly to keep history
    • Metadata too large
    • File-oriented architecture impacting performance
    • Poor data quality
    • Data duplication - hard to implement BI tasks, leading to two data copies: one in the lake, and another in a warehouse, often creating sync issues.
    • Requires heavy data ops infrastructure

    Data lakehouse: Merges the benefits of its predecessors. It has a transactional layer on top of the data lake that allows you to do both BI and data science in one platform. The data lakehouse cleans up all of the issues of the data lake, supporting structured, unstructured, semi-structured, and streaming data.

    Current Data Environments and Tools

    The following tools summary is from my deploying the tools as a CDO/CDAO and executive general manager, not as an architecture or engineer. This is a synopsis of the top-line features of each but if you want to add to your experience with the features please reply to the post and add to the synopsis.

    What is Snowflake?

    Snowflake is a highly flexible cloud-based big data warehouse that has some unique and specialized data security capabilities allowing businesses to transition their data to the cloud as well as to partner and share data. Snowflake has made much progress in building partnerships and APIs and integrations. One interesting possibility that marketers may want to consider is that snowflake can be leveraged as the CDP directly and activate campaign data through a number of their partners. See their website for more details.

    Snowflake is a data lakehouse that like its competitors is indifferent to structure variability and can support structured, semi-structured, and unstructured data. Its uniqueness for me is a few folds:

    • Ability to create highly secure data zones (a key strength) - You can set security at the field and user level. Strong partners like Alation and High Touch (a reverse ETL tool or ELT).
    • Ability to migrate structured and SQL-based databases to the cloud.
    • Ability to build unstructured data in the cloud for new data science applications.
    • Ability to use Snowflake in a variety of contexts as a CDP or a marketing subject area. If Snowflake becomes your CDP, you save the expense and other issues of having multiple marketing subject areas.

    Many organizations today are using data clouds to create a single source of truth. Snowflake can ingest data from any source, or format, using any method (batched, streaming, etc.), from anywhere. In addition, Snowflake can provide data in real time. Overall, it is good practice to have the marketing and analytics environments reside in one place such as Snowflake. Many times, as you generate insights you want to operationalize those insights into campaigns hence having them in one CDP environment improves efficiency. High-touch marketers, supported by their data analytics colleagues and Snowflake, can activate their data and conduct segmentation and analysis all in one place. Snowflake data clouds enable many other use cases:

    • One version of the truth.
    • Identity resolution can live in the Snowflake data cloud. Native integrations include Acxiom, LiveRamp, Experian, and Neustar.
    • You don’t have to move your data, so you increase consumer privacy with Snowflake. There are advanced security and PII protection features.
    • Clean room concept: No need to match PII to other data providers and move data. Snowflake has a media data cloud, so working with media publishers who are on Snowflake (such as Disney ad sales and other advertising platforms) simplifies targeting. As a marketer, you can work with publishers who built their business models on Snowflake without exposing PII, etc. Given the transformation that is happening due to the death of the third-party cookie, this functionality/capability could be quite impactful.

    What is Databricks?

    Databricks is a large company that was founded by some of the original creators of Apache Spark. A key strength of Databricks is that it’s an open unified lakehouse platform with tooling that helps clients collaborate, store, clean, and monetize data. Data science teams report the collaboration features were incredible. See the interview below with Ed Lucio.

    It supports data science and ML, BI, real-time, and streaming activities:

    • It’s software as a service with cloud-based data engineering at its core.
    • The lakehouse paradigm allows for every type of data.
    • No or low-performance issues.
    • Databricks uses a Delta Lake storage layer to improve data reliability, using ACID transactions, scalable metadata, and table-level and row-level access control (RLAC).
    • Able to specify the data schema
    • Delta Lake allows you to do SQL Analytics, an easy-to-use interface for analysts.
    • Can easily connect to PowerBI or Tableau.
    • Supports workflow collaboration via Microsoft Teams connectivity.
    • Azure Databricks is another version for the Azure Cloud.
    • Databricks allows access to open-source tools, such as ML Flow, TensorFlow, and more.

    Based on managing data scientists and large analytics teams, I would say that Databricks is preferred over other tools due to its interface and collaboration capabilities. But as always it depends on your business objectives in terms of which tool you select.

    What is DataRobot?

    DataRobot is a data science tool that can also be considered an autoML approach: it automates data science activities and thus furthers the democratization of machine learning and AI. The automation of the modeling process is excellent. This tool is different from Databricks which deals with data collection and other tasks. It helps fill the gap in skill sets given the shortage of data scientists. DataRobot:

    • Builds machine learning models rapidly.
    • Has very robust ML Ops to deploy models quickly into production. ML Ops brings the monitoring of models into one central dashboard.
    • Creates a repository of models and methods.
    • Allows you to compare models by methods and assess the performance of models.
    • Easily exports scoring code to connect the model to the data via an API.
    • Offers a historical view of model performance, including how the model was trained. (Models can easily be retrained.)
    • Includes a machine learning resource to manage model compliance.
    • Has automated feature engineering; it stores the data and the catalog.

    Using Databricks and DataRobot together helps with both data engineering AND data science.

    Now that we have a level set on the tools and vendors in the space, let’s turn to our interview with Ed Lucio.

    Interview With Ed Lucio


    People shake hands before an interview

    Tony Branda:

    Many firms struggle to deploy machine learning and data operations tools in the cloud and to get the data needed for data science into the cloud. Why is that? How have you seen firms resolve these challenges?

    Ed if you could unpack this one in detail? Thanks, Tony.

    Ed Lucio:

    From my experience, the challenge to migrate data infrastructures and deploy cloud-based advanced analytics models is a more common issue in traditional/larger organizations which have at least one of the following: built significant processes on top of legacy systems, is in vendor/tooling lock-in, in a ‘comfortable’ position where cloud-based advanced analytics adoption is not the immediate need, and the information security team is not yet well adept on how this modern technology is aligned to their data security requirements.

    However, in progressive and/or smaller organizations where there is an alignment from senior leaders down to the front line (coupled with the immediate need to innovate), cloud-based migration for infrastructures and deployment of models is almost natural. It is scalable, more cost-effective, and flexible enough to adjust to dynamic business environments.

    I have seen some large organizations resolve these obstacles through strong senior leadership support where the organization starts building cloud-based models and deploys smaller use cases with less critical components for the business. The objective is just to prove the value of the cloud first, then once a pattern has been established, the company can scale up to accommodate bigger processes.

    Tony Branda:

    Why is Databricks so popular as one tool category (Data Ops/ML Ops), and what does their lakehouse concept give us that we could not get from Azure, AWS, or other tools?

    How does Databricks help data scientists?

    Ed Lucio:

    What I personally like about Databricks is the unified environment that helps promote collaboration across teams and reduces overhead when navigating through the “data discovery to production” phases of an advanced analytics solution. Whether you belong to the Data Science, Data Engineering, or Insights Analytics team, the tool provides a familiar interface where teams can collaborate and solve business problems together. Databricks provide a smooth flow from managing data assets, performing data exploration, dashboarding, and visualization, to machine model prototyping and experiment logging, and code and model version control. When the models are deemed to be ready by the team, deploying these through a job scheduler and/or an API endpoint is just a couple of clicks away and flexible enough for the business needs whether it is needed for either batch or real-time scoring. Lastly, it is built on open-source technology which means that when you need help, the internet community would almost always have an answer (if not, your teammates or the Databricks Solutions Architect would be there to assist). Other sets of cloud tools would be able to provide similar functionalities, but I haven’t seen one as seamless as Databricks.

    Tony Branda:

    On Snowflake, AutoML tools, and others, how do you view these tools, and what is your view on best practices?

    Ed Lucio:

    Advanced analytics discovery is a journey where you have a business problem, a set of data and hypotheses, and a toolkit of mathematical algorithms to play with. For me, there is no “unicorn” tool (yet) in the market able to serve all the data science use-case needs for the business. Each tool has its own strengths, and it would need some tinkering on how every component would fit the puzzle in achieving the end business objective. For instance, Snowflake has nice features for managing the data asset in an organization, while the AutoML tooling (DataRobot/H2O) is good for automated machine learning model building and deployment.

    However, even before proceeding to create an ML model, an analyst would need to explore the dataset for quality checks, understand relationships, and test basic statistical hypotheses. The data science process is iterative, and organizations would need the tools to be linked together so that interim outputs are communicated to the stakeholders to pivot or confirm the initial hypothesis, and to share the outputs with the wider business to create value. Outputs from each step would typically need to be stitched together to get the most from the data. For example, a data asset could be fed into a machine learning model where the output would be used in a dashboarding tool for reporting. Then, the same ML model output could be further enhanced by business rules and another ML model to be fit for purpose on certain use cases.

    On top of these, there must be a proper change control environment governing the code and model versioning and transitioning of codes/models from development, pre-prod, and prod environments. After deploying the ML model, it needs to be monitored to ensure that model performance is within a tolerable range, and the underlying data has not drifted from the training set.

    Tony Branda:

    Are there any tips and tricks you’d recommend to leaders in data analytics (DA) or data science (DS) to help us evaluate these tools?

    Ed Lucio:

    Be objective when evaluating the data science tooling and work with the data science and engineering teams to gather requirements. Design the business architecture which supports the organization’s goals, then work backward together with the enterprise architect and platform team to see which tools would enable these objectives. If the information security team objects to any of the candidate toolings, have a solution-oriented mindset to find alternative configurations to make things work.

    Lastly, strong support from the senior leadership team and business stakeholders is essential. Having a strong focus on the potential business value, the need to enable the data science tools would always come in handy.

    Tony Branda:

    What is the difference between a data engineer and a data scientist, and an ML engineer (in some circles referred to as a data science data engineer)? Is it where they report or do they have substantial skill differences? Should they be on the same team? How do we define roles more clearly?

    Ed Lucio:

    I see the data engineers, ML engineers, and data scientists being part of a wider team working together to achieve a similar set of objectives: to solve business problems and deliver value using data. Without going into too much detail:

    • Data engineers build reliable data pipelines to be used by insight analysts and data scientists.
    • Data scientists experiment (i.e., apply the scientific process) and explore the data on hand to build models addressing business problems.
    • ML model engineers work collaboratively with the data scientists and data engineers to ensure that the developed model is consumable by the business within the acceptable range of standards (i.e., batch scoring vs real-time? Will the output be surfaced in a mobile app? What is the acceptable latency?).

    Each of these groups would have their own sets of specialized skills, but at the same time, should have a common level of understanding of how each of their roles work side-by-side.

    Many thanks to Ed Lucio, Senior Data Scientist at Spark in New Zealand, for his contributions to this article.

    In summary, this article has provided a primer on data lakehouses and three of what I consider to be the leading-edge tools in the cloud data lakehouse and machine learning space. I hope Ed Lucio’s POV on the tools and their importance to data science was helpful to those considering their use. At the end of the day, all of this—selection of environments and tools—depends on the business needs and goals: what are the problems that need solving, as well as the level of automation the business is driving towards.

      As always, I’d love to hear about your experiences. What has your experience been with data lakehouses, ML Ops, and data science tooling? I look forward to hearing from you regarding this post.

      Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

      The Data Science ROI Of Moving To …

      Competing With Spreadsheets — The Joy And The Horror!



      Bill Gates has a lot to answer for. Every office computer has a spreadsheet application on it. Any task involving numbers is done on a spreadsheet.

      Selling contact center technology, I am frequently told, “No problem, they’re only using spreadsheets.”

      A professional colleague told me that when people replace existing technology, either they are looking for a solution that does more for the same price that they are already paying, or for something that does the same as their current solution, but for a lower price.

      Since everybody’s computer already has a spreadsheet application on it, they see it as a zero-cost application.

      How do you compete against a zero-cost application?

      Know Your Enemy


      Microsoft Excel icon on a laptop

      Sun Tzu wrote: “If you know the enemy and you know yourself, you need not fear the result of 100 battles.”

      Now we’ve identified spreadsheets as our “enemy,” what are their strengths and what are their weaknesses?

      The Strengths Of Spreadsheets


      Man on laptop looks at a spreadsheet

      Their greatest strength is that they are essentially free. Even if a company does buy your solution, they are not going to give up their spreadsheet licenses. They use spreadsheets for so many other parts of the business, so there is no saving by adopting your product.

      Spreadsheets are the legacy solution. People have designed and amended their files over the years to meet their needs exactly. People feel as comfortable using spreadsheets as they would wearing their favorite pair of slippers.

      The final strength of spreadsheets is their flexibility. The average Excel user regularly uses about 5% of its full functionality. Spreadsheets’ formulae address all aspects of mathematics and contain many tools for analyzing and processing data. Most software cannot emulate all the different functions that spreadsheets have to offer.

      The Holes In The Spreadsheets’ Armor


      Empty spreadsheet on Microsoft Excel

      Spreadsheets were originally designed to make calculations and save them in files. These calculations could be shared by sharing the file.

      More than one person can work on a spreadsheet at the same time, but it has its challenges. The key challenge is that the number of users working on a spreadsheet at any one time is limited for practical purposes, to two or three. If you create a database table in Excel, it’s not possible to have 100 users making real-time changes to it simultaneously.

      Security is another headache. Spreadsheets can be password protected, but that password protection cannot be integrated into an identity and access management (IAM) tool. The password is unique to the spreadsheet and not to the user If 100 users use the same spreadsheet, then 100 users know the same password. Alternatively, each user can use their own spreadsheet and then regularly merge the files. This requires a lot of manual work, which is prone to error.

      Automation is possible within spreadsheets, but it’s complicated. Not many people know how to write macros in VBA. If your product has a functionality that would need a macro to replicate it, then it’s unlikely that your customer’s organization will have someone who can write that macro. On the other hand, if your customer’s spreadsheets use macros to perform certain functions, it’s pretty unlikely that your product will perform the same function as the macro. I have often worked with customers using spreadsheets with macros. When I ask how they work, they will tell me that the person who wrote the spreadsheet has moved on. No one in their company knows how it works, but they have to keep using it.

      Replacing Spreadsheets — Including The Value In (Return On Investments) ROI Calculations


      Woman looks at a spreadsheet on her computer

      As discussed, people look for solutions that do more for the same price that they are already paying, or for solutions that do the same as their current solutions, but for a lower price.

      The trick is to understand what are the units of work being done on the spreadsheet and to attribute a financial value to them based on the manual labor. This is usually based on the time it takes to do the work. If you can show that your product can do the same units of work in a shorter period of time, or with the reduced attention of a human worker, then you can estimate a cost saving.

      This manual work can include the costs of someone checking the work, and also the costs of rework to rectify mistakes.

      The second way to calculate the ROI is to look at the consequences of non-compliance. Keeping customer data on a spreadsheet is not very secure for the reasons mentioned above. It is likely to violate data protection laws. You can also factor the likely penalties, fines, and legal costs into the ROI.

      The wrap-up

      If you’re selling to small and medium-sized customers, spreadsheets can be a fearsome incumbent to beat. You need to reveal the effect of the spreadsheet’s weaknesses on the business while being ready to handle objections based on the spreadsheet’s strengths.

      Are you competing against spreadsheets? Why are your customers considering replacing them? How are spreadsheets holding them back? What do they hope that your product can do that spreadsheets can’t? How does the value your product creates compare to the value spreadsheets generate?

      Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

      Competing With Spreadsheets — The Joy And …

      Data Never Sleeps. How Actionable Is Your Reporting?



      The business seems to be doing better, but you have reporting to show how well it's actually doing? You want to collect data and turn it into information. This allows the business to make decisions based on actionable reporting. How much business intelligence (BI) does your organization have?

      COTS Applications For Actionable Reporting


      Most COTS (commercial-off-the-shelf) applications will have standard/canned reports. These are the “common” reports used by customers. If you’re implementing a new system and there are many standard reports, start with 10 top reports so you don’t overwhelm the end users. If you’re not sure what these are, ask the vendor to recommend them (especially if you’re changing your process and following the vendor’s best practice). Then the end users can utilize other standard reports as they get more comfortable and familiar with the application.

      For example, if you just purchased a new phone system for the call center, find the reports that detail measurements such as the number of calls, dropped calls, wait time, peak times, etc.

      Identify and define key performance indicators (KPIs), which are measurable values to analyze for better decision making. KPIs may be related to a specific function, line of business, or group of individuals. You can take a group of KPIs and create a dashboard. For example, some KPIs related to the help desk are:

      • Tickets by status (e.g., open, on hold, completed, canceled)
      • Tickets assigned by technician
      • Average resolution time
      • Rate of first-time resolutions
      • Customer satisfaction

      Once you’re comfortably leveraging the standard reports, you may want to ask the vendor (or your team’s developer/programmer) to customize standard reports or create custom reports with additional specific information you’re looking for. This may involve combining data from multiple tables within the application.

      Two typical ways to serve these reports: 1) “push” reports to the end users on a scheduled frequency such as via email daily; or 2) have the end user “pull” reports which gives them the ability to access the reports as needed.

      But reporting isn’t limited to COTS applications. You can create remarkable reports using Excel such as PivotTables, Pivot Charts, or Slicers.

      Other Tips & Tools For Actionable Reporting


      Another phenomenal visualization tool that is part of Office 365 is Power BI. It’s part of the Microsoft family (similar to Excel) so it’s straightforward to use. You can create a dashboard visualizing year-over-year comparisons, market segmentations, drill downs, etc.

      If you’re new to Power BI, you can download the free desktop version (which admittedly does have limitations). If you find that Power BI does meet your basic reporting needs and you want more, then you can upgrade to the Pro license.

      If you have enormous amounts of data, you may want to consider creating a data lake (raw data) or data warehouse (structured data). You can pull in data from multiple disparate systems into one central repository. There are ETL (extract transform load) tools to load the data into the data warehouse. You can cube the data so that the end users can roll up, drill down, and slice the data, do “what if” scenarios, etc.

      TIP: If you allow end users to create their own reports, it’s critical that they understand the various data fields. Are you able to give them some type of data dictionary, or are they able to certify the results? It would be disastrous if they selected the wrong date or revenue field which produced incorrect results and sent the report to the board of directors.

      Other reporting considerations:

      1. Have a separate reporting database so that you don’t affect the production database. You don’t want an end user to kick off a complex query inadvertently causing production response to come to a crawl or even freeze.

      • If you do set up a reporting database you need to decide how frequently the data will be updated—real time, once in the evening, etc.

      2. Data needs to be secured appropriately (security). Who should have access to specific data fields should be determined by the data owner. For example, only a small handful of people should be able to access salary/payroll information.

      3. Data should be classified such as public, internal, confidential, or restricted. Access should be controlled as appropriate.

      4. Create a customer portal (extranet) to empower your customers (also partners and vendors) so that they can pull their own reports and information.

      There is a quote by Lois Horowitz: “Not having the information you need when you need it leaves you wanting. Not knowing where to look for that information leaves you powerless. In a society where information is king, none of us can afford that.”

      For more information on actionable reporting, follow me on LinkedIn!

      Acquiring Job Advance Your Career Career Changing Careers and Employment Education Internship

      Data Never Sleeps. How Actionable Is Your …