This is a practical book, to be sure, but it is also a book about hope and posi- tive change. I am quite sincere. The delivery of electricity is deeply rooted in the principle of universal access; when clean, reliable energy is available it contributes to poverty alleviation, improved social conditions, and enhanced economic development. In the developed world, we know this to be true. The digital fabric of our lives is a testimony to the importance of energy security. Across the globe, we have seen the vital contributions that electrification has brought to the development of economies and an enhanced quality of life. Nonetheless, this supreme engineering achievement has languished, and we are deeply challenged.
Modern electrification systems are degrading and inefficient in myriad ways, yet the complex and difficult operating conditions of the energy business have been slow to adapt and advance to improve these circumstances. However, with the advent of the information-enabled, two-way grid, we have an opportunity to meet these challenges directly. It is the thesis of this book that through the application of big data analytics and subsequent improvements in situational awareness of the millions of miles of grid across the world, we will be able to integrate renewable energy systems, introduce economic and operating efficien- cies, and bring energy services to the over 1 billion people across the world who have no electricity. It is also the view of this work that utilities are confronted with a very difficult charge indeed—to evolve rapidly towards a business stand- point that capitalizes on these key technologies. It is going to take a resolute effort from technologists, utility stakeholders, political bodies, and energy con- sumers to protect and improve the performance of the grid, as well as affect the change necessary to shield our economies and defend the environment.
I hope to shed light on the considerable potential that big data analytics brings to the electrical power system by virtue of a fully realized analytics strategy. The unprecedented access to the immense and growing volumes of data now available to describe the electrical system and its consumers can pro- vide powerful and nearly instantaneous insight. This insight not only improves the ability to optimize day-to-day operations, but in times of stress, is the core enabler of effective decision-making and critical communications. When faced with uncertain conditions of extreme weather, terrorism, or other disasters, the safety and continuity of reliable energy delivery is without measure.
One could hardly say that the grid is broken today, but system reliability and efficiency has degraded over the past several decades. And change has been slow to come. It is imperative that, as a society, we find ways to make the grid more resilient, secure, efficient, reliable, and capable of integrating with the lives of consumers. The technical innovations inherent in big data analytics for the smart grid are the first step and the future step.

Dedication v

Contents vii

Foreword xv

Preface xxi

About the Author xxiii

Acknowledgments xxv

Section One The Transformative Power of Data Analytics

Chapter One: Putting the Smarts in the Smart Grid 3

      1.1 Chapter Goal 3

      1.2 The Imperative for the Data-Driven Utility 4

      1.3 Big Data: We'll Know It When We See It 7

      1.4 What Are Data Analytics? 8

      1.4.1 The Data Analytics Infrastructure 9

      1.5 Starting from Scratch 11

      1.5.1 Mind the Gap 12

      1.5.2 Culture Shift 12

      1.5.3 A Personal Case Study 13

      1.5.4 Ouija Board Economics 15

      1.5.5 Business as Usual Is Fatal to the Utility 17

      1.5.6 To Be or Not to Be 18

      1.6 Finding Opportunity with Smart Grid Data Analytics 19

Chapter Two: Building the Foundation for Data Analytics 21

      2.1 Chapter Goal 21

      2.2 Perseverance Is the Most Important Tool 22

      2.2.1 "It's Too Hard" Is Not an Answer 23

      2.3 Building the Analytical Architecture 23

      2.3.1 The Art of Data Management 25

      2.3.2 Managing Big Data Is a Big Problem 25

      2.3.3 The Truth Won't Set You Free 25

      2.3.4 One Size Doesn't Fit All 28

      2.3.5 Solving the "Situation-Specific" Dilemma 29

      2.3.6 The Build-Versus-Buy War Rages On 30

      2.3.7 When the Cloud Makes Sense 32

      2.3.8 Change Is Danger and Opportunity 35

Chapter Three: Transforming Big Data for High-Value Action 37

      3.1 Chapter Goal 37

      3.2 The Utility as a Data Company 38

      3.2.1 Creating Results with the Pareto Principle 39

      3.3 Algorithms 40

      3.3.1 The Business of Algorithms 41

      3.3.2 Data Classes 41

      3.3.3 Just in Time 43

      3.4 Seeing Intelligence 44

      3.4.1 Remember the Human Being 46

      3.4.2 The Problem with Customers 46

      3.4.3 The Transformation of the Utility 49

      3.4.4 Bigger Is Not Always Better 49

      3.5 Assessing the Business Issues 51

      3.5.1 Start with a Framework 51

Section Two The Benefits of Smart Grid Data Analytics

Chapter Four: Applying Analytical Models in the Utility 57

      4.1 Chapter Goal 57

      4.2 Understanding Analytical Models 58

      4.2.1 What Exactly Are Models? 61

      4.2.2 Warning: Correlation Still Does Not Imply Causation 62

      4.3 Using Descriptive Models for Analytics 63

      4.4 Using Diagnostic Models for Analytics 64

      4.4.1 How Diagnostic Tools Help Utilities 65

      4.5 Predictive Analytics 65

      4.6 Prescriptive Analytics 67

      4.7 An Optimization Model for the Utility 69

      4.8 Toward Situational Intelligence 69

Chapter Five: Enterprise Analytics 73

      5.1 Chapter Goal 73

      5.2 Moving Beyond Business Intelligence 74

      5.2.1 Energy Forecasting 75

      5.2.2 Asset Management 75

      5.2.3 Demand Response and Energy Analytics 77

      5.2.4 Dynamic-Pricing Analytics 84

      5.2.5 Revenue-Protection Analytics 87

      5.2.6 Breaking Down Functional Barriers 88

Chapter Six: Operational Analytics 91

      6.1 Chapter Goal 91

      6.2 Aligning the Forces for Improved Decision-Making 92

      6.3 The Opportunity for Insight 93

      6.3.1 Adaptive Models 94

      6.4 Focus on Effectiveness 94

      6.4.1 Visualizing the Grid 96

      6.5 Distributed Generation Operations: Managing the Mix-Up 98

      6.6 Grid Management 100

      6.6.1 The Relationship Between Standards and Analytics 102

      6.7 Resiliency Analytics 102

      6.8 Extracting Value from Operational Data Analytics 104

Chapter Seven: Customer Operations and Engagement Analytics 107

      7.1 Chapter Goal 107

      7.2 Increasing Customer Value 108

      7.2.1 Customer Service 108

      7.2.2 Advanced Customer Segmentation 109

      7.2.3 Sentiment Analysis 110

      7.2.4 Revenue Collections 112

      7.2.5 Call Center Operations 113

      7.2.6 Utility Communications 114

      7.3 What's in It for the Customer? 117

      7.3.1 Enhanced Billing and Customer-Facing Web Portals 118

      7.3.2 Home Energy Management 120

      7.3.3 Strategic Value 121

Chapter Eight: Analytics for Cybersecurity 123

      8.1 Chapter Goal 123

      8.2 Cybersecurity in the Utility Industry 124

      8.2.1 The Threat Against Critical Infrastructure 124

      8.2.2 How the Smart Grid Increases Risk 127

      8.2.3 The Smart Grid as Opportunity for Dark Mischief 128

      8.3 The Role of Big Data Cybersecurity Analytics 129

      8.3.1 Predict and Protect 131

      8.3.2 Cybersecurity Applications 133

      8.3.3 Proactive Approaches 134

      8.3.4 Global Action for Coordinated Cybersecurity 134

      8.3.5 The Changing Landscape of Risk 136

Section Three Implementing Data Analytics Programs for Sustained Change

Chapter Nine: Sourcing Data 141

      9.1 Chapter Goal 141

      9.2 Sourcing the Data 142

      9.2.1 Smart Meters 143

      9.2.2 Sensors 145

      9.2.3 Control Devices 146

      9.2.4 Intelligent Electronic Devices 147

      9.2.5 Distributed Energy Resources 148

      9.2.6 Consumer Devices 149

      9.2.7 Historical Data 150

      9.2.8 Third-Party Data 151

      9.3 Working with a Variety of Data Sources 152

      9.3.1 Data Fusion 152

Chapter Ten: Big Data Integration, Frameworks,and Databases 155

      10.1 Chapter Goal 155

      10.2 This Is Going to Cost 156

      10.3 Storage Modalities 157

      10.3.1 Hyperscale 157

      10.3.2 Network-Attached Storage 157

      10.3.3 Object Storage 158

      10.4 Data Integration 158

      10.5 The Costs of Low-Risk Approaches 159

      10.6 Let the Data Flow 160

      10.6.1 Hadoop 161

      10.6.2 MapReduce 162

      10.6.3 Hadoop Distributed File System 164

      10.6.4 How Does This Help Utilities? 164

      10.7 Other Big Data Databases 165

      10.7.1 NoSQL 166

      10.7.2 In-Memory or Main Memory Databases 167

      10.7.3 Object-Oriented Database Management Systems 167

      10.7.4 Time Series Database Servers 168

      10.7.5 Spatial and GIS Databases 168

      10.8 The Curse of Abundance 169

Chapter Eleven: Extracting Value 171

      11.1 Chapter Goal 171

      11.2 We Need Some Answers Here 172

      11.2.1 How Long Does This Take? 173

      11.3 Mining Data for Information and Knowledge 176

      11.4 The Process of Data Extraction 177

      11.4.1 When More Isn't Always Better 181

      11.4.2 Running for Performance 182

      11.4.3 Hadoop: A Single-Purpose Batch-Data Platform? 182

      11.5 Stream Processing 183

      11.5.1 Complex Event Processing 184

      11.5.2 Process Historians 185

      11.6 Avoid Irrational Exuberance 186

Chapter Twelve: Envisioning the Utility 187

      12.1 Chapter Goal 187

      12.2 Big Data Comprehension 188

      12.3 Why Humans Need Visualization 189

      12.3.1 Walking Toward the Edge 190

      12.4 The Role of Human Perception 192

      12.4.1 Preattentive Processing 193

      12.5 The Utility Visualized 197

      12.5.1 Advancing Business Intelligence 200

      12.5.2 High-Impact Operations 200

      12.5.3 Improving Customer Value 201

      12.6 Making Sense of It All 203

Chapter Thirteen: A Partnership for Change 205

      13.1 Chapter Goal 205

      13.2 With Big Data Comes Big Responsibility 206

      13.2.1 Abandon All Hope, Ye Who Enter Here? 207

      13.3 Privacy, Not Promises 208

      13.3.1 Consent 209

      13.3.2 Data Management 211

      13.3.3 Governance 212

      13.4 Privacy Enhancement 213

      13.4.1 Enabling Consent 214

      13.4.2 Data Minimization 215

      13.4.3 The Role of Metadata 216

      13.5 The Utility of the Future Is a Good Partner 216

Glossary 219

Index 225

Carol L. Stimmel began working with "big data analytics" in 1991 while hack- ing code and modeling 3D systems for meteorological research—years before that combination of words ever became buzzword compliant. In those 23 years, she has spent the last 7 focusing on the energy industry, including smart grid data analytics, microgrids, home automation, data security and privacy, smart grid standards, and renewables generation. She has participated in emerg- ing technology markets for the majority of her career, including engineering, designing new products, and providing market intelligence and analysis to utili- ties and other energy industry stakeholders.PA\Carol has owned and operated a digital forensics company, worked with cutting-edge entrepreneurial teams; co-authored a standard text on organiza- tional management, The Manager Pool; and held leadership roles with Gartner, E Source, Tendril, and Navigant Research. She is the founder and CEO of the research and consulting sustainability company, Manifest Mind, LLC, which brings rigorous, action-based insight to advanced technology projects that create and maintain healthy ecosystems for people and the environment. Carol holds a BA in Philosophy from Randolph-Macon Woman's College.

中科院文献情报中心