Mark Weatherford, California's CISO, transformed the government into an enterprise organization, Illena Armstrong reports.
Not too long ago, the information security challenges facing the state of California were about as substantial as the state itself, especially since its government offices were far from being the model of high-tech.
Among Governor Arnold Schwarzenegger's goals after assuming the Golden State's top office in 2003, was “to change the way technology was done in California,” says the state's CIO Teri Takai, a cabinet level appointee of the governor. “California had not been known as doing technology well.”
Even though it boasts of Silicon Valley – a region that many view as the top technological economic center in the country – California's government was far behind most IT innovations and practices of the day. And, information security initiatives, too, needed some vital upgrading.
“One of the challenges is that California is just huge,” says Takai. A major trial that goes along with this is "to ensure that security is really one of the cornerstones of all of the transformations we're doing here.”
So, when she began recruiting for the first centralized CISO of California – after joining as CIO herself in 2008 – it was with a long list of requirements in mind. She wanted to find a professional who was nationally recognized, had the capabilities to deal with the operational side of information security, had experience working closely with other departments and their directors, and had the aptitude to build alliances internally, at federal levels, with law enforcement bodies and many others. Plus, of course, she needed someone proficient at crafting state public policies and practices that addressed both internal and external, or customer-facing, security needs.
Mark Weatherford, who is 2010's winner of the SC Awards CSO of the Year, had these varied experiences and a whole lot more. His skills and experience have proven indispensible since taking on the post of director and chief information security officer of California in 2008. Moving from the same seat in Colorado, he already knew some of the trials that go along with a state gig. When Governor Bill Ritter took office in 2007, many who were appointed by his predecessor changed. However, Weatherford was reappointed.
“That doesn't happen all that frequently,” says Colorado's CIO Michael Locatis. “He's a star.”
While in Colorado, he worked with that state's legislature to get laws passed to formally establish a security program for the first time in the state, says Locatis. By establishing and getting approval for a state statute addressing information security, he ensured it had “staying power,” adds Locatis. It also demonstrated his skill in getting consensus among state legislators – a rare trait.
Too, he supported Ritter's plan for consolidation of IT in Colorado to help establish an overall enterprise delivery capability. With over 32,000 employees, 64 counties and over 3,000 endpoints statewide, this was (and still is) no easy task. Weatherford, though, played a critical role in writing consolidation legislation that passed almost unanimously in 2008, and he worked tirelessly on various projects to ensure that IT and cybersecurity always were integrated.
Alan Paller, director of research for the SANS Institute, says Weatherford has proven his proficiency in pushing security forward in state environments where many other CISOs give up or just become compliance dictators, often because of lack of funding, staff and authority.
In his current California position, among many other accomplishments he has established the first official IT security plan and strategy for the state, created an extensive security policy to address social media usage, implemented regulations for remote and traveling workers, and begun the extensive task of standardizing IT operations statewide – a move crucial in meeting overall consolidation of sundry technologies, says Takai. As well, he has taken on the lead role of transitioning the state's decentralized IT operations into a centralized business based on a federated model, she says. The state has about 10,000 IT employees, 1,200 of whom now manage the core of the state's IT programs and establish policies for the rest of the IT organizations within various state departments. And information security officers who didn't report into one state CISO prior to 2008, but rather reported into individual department leaders – from CIOs to many others – now are overseen by Weatherford.
These massive developments to consolidate an extremely decentralized environment and staffing structure led to the receipt of a $4 million grant from the Department of Homeland Security (DHS), specifically for information security. And this is a good thing given the massive deficit confronting the state, which is estimated to hit about $20 billion this year.
Paller notes that overall budget shortfalls in Colorado and then California led to Weatherford and his staff losing about 15 percent of their salaries.
“But Mark didn't back off or become a curmudgeon. He went out and found resources, tapping into DHS funds at a level that is unparalleled in most other states and using it to facilitate improvement, getting other resources into the state by serving as a test bed for a cool project on cyber-inventory and risk determination, and even bringing in a grant to develop cybercamps to nurture young hackers who can become cyberguardians and cyberwarriors,” says Paller. “But, even more important to me is that he finds ways of improving security impressively without spending a lot of money, by investing in small ways. It's innovation through imitation. Others spend way too much reinventing the wheel. My favorite example is the most current: He is the only state CISO who has taken up the powerful new security innovation pioneered at the State Department -- provable 90 percent reduction in cyber risk and more coming fast.”
With the California state budget dedicating about $3 billion specifically for overall IT infrastructure, which obviously includes information security, Takai adds that money, indeed, must be spent wisely.
“It's how we use the $3 billion in a way that improves security, as well as operations,” she says, explaining further that security cannot be independently pursued from IT projects, but simply should be and will be a standard part of all of California's IT projects.
“That change of attitude and role of being a watchdog to being up front and collaborative is critical,” she explains. “By having the varied experiences that he's had, Mark has a great perspective. He has done just a fabulous job since he came here.”
In an interview with SC Magazine, Weatherford was able to explain a bit more about his accomplishments in California that made possible his being selected as CSO of the Year. Here he shares some steps he has taken to bolster security plans that possibly could translate to other environments in which SC Magazine readers find themselves.
Illena Armstrong: How long have you been in information security?
Mark Weatherford: I was in the cryptologic community in the U.S. Navy for more than 20 years and wrote my graduate thesis on information security at the Naval Postgraduate School in 1994. I've had information security responsibilities in every job since then.
IA: Can you highlight the positions and organizations that helped you prepare for your stint for the state of California?
MW: The easy answer is all of them, but my biggest operational role was as the operations officer at the Fleet Information Warfare Center (FIWC) from 1999-2001, where we had worldwide responsibility for network security monitoring and the Naval Computer Incident Response Team (NAVCIRT). We also started the Navy's Red Team operations, where we would perform penetration testing against Navy sites.
After leaving the Navy, I joined the Raytheon company where I was responsible for building out and operationalizing the Security Operations Center (SOC) for the U.S. Navy/Marine Corps Intranet (NMCI) in San Diego. NMCI is a massive program that consolidated all naval network and information assurance operations at three central network operations facilities in the United States. As a brand new organization, we built out the physical infrastructure, developed all security policies, implemented operational security measures and configuration management practices, and established an enterprise incident response program and all supporting procedures.
My previous job as the CISO for the state of Colorado was my first foray into state government, so I learned a lot about how government actually works. It's a good thing a civics exam wasn't part of the interview process or I would have failed. I'm a lot wiser now. When Governor Bill Owens brought me in as the first state CISO, the enterprise security program didn't exist. I inherited an empty desk without a chair. Fortunately, I had some great support behind the scenes and was able to get some pretty significant funding from the governor to kick off a few initiatives, including an enterprise risk assessment and a statewide laptop encryption project – that both identified weaknesses in some of the state's critical systems and also mitigated a big problem with lost and stolen laptops.
IA: What about pertinent training and certifications?
MW: I hold the CISSP and CISM certifications and have attended several SANS security courses.
IA: Any mentors who really helped you over the years to get to this point of understanding about information security?
MW: Alan Paller from the SANS Institute is a great friend and has provided me with sound guidance over the years. He's devoted his life to making a difference in a lot of people's lives, mine included, and it's no exaggeration to say that he has shaped the way many security professionals think about information security. Alan is the kind of person who leaves a mark in the world and is a great example.
The people I've learned the most from actually worked for me, and I've had the opportunity to work with some incredibly talented people over the years. Michele Robinson from the state of California, Scott King from Sempra Energy, Mike Weber from Critigen, Travis Schack from the state of Colorado, and Mike Sheldon from Smartronix are true alpha-geeks and part of my “security dream team.” With these five people, I think I could conquer the world … perhaps one of these days?
IA: What have been your major achievements in the last year of which you're most proud (and likely helped you receive this recognition)?
MW: The Information Security Policy Refresh project is one thing that I'm extremely proud of because it will have such a lasting impact on how we approach information security across the state enterprise. As we began this project, I underestimated its difficulty and learned that issuing policy in government is decidedly non-trivial, even for no-brainer issues like security governance. California government is a vast organization, employing over 225,000 people distributed across more than 130 agencies, departments, offices, boards and commissions, each with their own CIO and information security officer (ISO). When I first arrived in June of 2008 and began making my rounds meeting the CIOs and ISOs, the one comment I heard over and over was, “We need an enterprise security policy framework to guide our organizations and help us justify our security requirements.” I've always believed that policy drives security, so this project became one of my highest strategic priorities. It's an on-going project, but we are steadily making progress and, when complete, it will have a profound effect on the ability of individual organizations to achieve consistency in their security posture.
Another project that has had a big impact is the publication of our “Enterprise Security Strategic Plan” last November. One of the things that IT and security organizations are good at is meeting the daily tactical requirements of keeping the engines humming. Conversely, one of the things these same organizations aren't always so good at is taking a strategic view of where they need to be headed. We spent about nine months developing our “Enterprise Security Strategic Plan.” It sets some realistic goals that will help us, as an enterprise, begin to get in front of some of our pressing security issues.
Because the funding and procurement process in California is so drawn out, I needed to find financial sources outside the state's normal general fund for some of the major projects I had in mind. The [Department of] Homeland Security Grants program was my key. My team and I developed nine grant applications for cybersecurity-related projects and we were awarded $3.7 million for two of them. We are also sharing in a $4 million grant with three other states in a Community Cyber Security Maturity Model (CCSMM) program where we are helping local governments with cybersecurity planning and awareness. This is more grant funding in one year than the state has received for cybersecurity in the past eight years combined.
IA: What were the major challenges associated with these?
MW: Because our enterprise information security program is so new, there wasn't a lot of history with projects like these, so we are establishing our niche as we mature. Getting credibility has been the key, but by engaging people from various state agencies in almost everything we do, I think we've been able to show that building an enterprise security program – with a true desire to raise the security posture of the state – is a worthwhile and noble effort that people want to be a part of. I've been able to recruit talented volunteers from a number of state departments for my projects simply because they want to help and be part of something that benefits the entire state. We have very dedicated security professionals in California state government.
IA: Who in your organization helped with these achievements?
MW: Everyone! As I mentioned earlier, we have 130 information security officers in California state government who, each in their own right, have tremendous responsibilities. I wish I had room and time to call each of them out individually. Some of them, based on the size of their departments, have more overall security responsibilities than I previously had as the CISO of the state of Colorado. In addition to the state ISO community, I'm very fortunate to have my own staff of outstanding security professionals who look at every overwhelming problem as simply another opportunity to excel. The funny thing about an award like this is that while I'm getting the recognition as CSO of the Year, it's really the people who do the heavy lifting every day that deserve the credit for our successes.
IA: Do you get enough support from your colleagues and bosses?
MW: I'm one of the truly lucky CISOs whose bosses get it and understand why we need a sound and active information security program. Both Governor Schwarzenegger and my boss, Teri Takai, have been incredibly supportive as I've made changes to the overall information security program in California state government. That doesn't mean I always get everything I want, but I'm not complaining either.
IA: What steps do you find integral in getting and maintaining such support?
MW: Under-promising and over-delivering.
IA: When you're undertaking various projects, do you have to work with managers of various business units?
MW: Yes, but I still find that I spend most of my time with departments CIOs helping them bridge the IT and security gap with the business units. A big part of my job is explaining risk and providing understandable explanations for why certain things should be done in certain ways.
IA: What advice would you give to your peers to get the funding, resources and support they need – especially in this still trying economy?
MW: First off, it's important that you don't cheat yourself in the requirements development of your planning. It's critical to identify those resources necessary to support security training, security tools and the ability to adhere to standards. Oftentimes, projects are established with a cost threshold in mind before the requirements are developed. This leads to cost-cutting and one of the first things to go is training. That's risky business and rarely ends up with everyone smiling.
We've figured out ways to get very creative with our training at drastically reduced rates by working with our training partners to host “beta” courses that haven't yet been approved for primetime. We do vendor “lunch and learns” on products we already have in the environment, encourage attendance at free webinars, and we also have our own staff experts conduct training on their areas of expertise. Most organizations already have internal expertise they can capitalize on if you look hard enough.
On the security tool side, there are dozens of excellent open source security tools that are industry de facto standards which many organizations depend on to monitor and identify vulnerabilities within their IT environments. We recently published a policy letter that formally establishes the use of open source software (OSS) in California state government as an acceptable practice. This policy simply formalized what is already a routine practice, but it has been received very warmly.
IA: Is there an ideal hierarchical structure when it comes to ensuring IT security is being addressed adequately in a corporate environment?
MW: This is an issue my colleagues and I have spent a lot of hours talking about over the years. In a perfect world, I think it's realistic to see the CISO as a peer of the CIO. Unfortunately, we don't live in a perfect world and I just don't think most organizations are mature enough for that. Because the CIO has overall responsibility for organizational IT, if the CISO is out of sync and driving security decisions contrary to the CIO's lead, it can lead to incredible confusion and frustration. This is only my opinion, but it just seems to work better when the CIO and CISO work together in a senior-subordinate relationship. Fortunately, I've always been fortunate to get along well with my bosses and feel like I'm given adequate opportunity to make my case for security. At the end of the day, and I still have to remind myself of this occasionally, we need to remember that the business units don't exist to help us meet our security goals, but rather the other way around. At times there may be business complexities that drive decisions in a direction we may not like.
IA: What about budgetary needs?
MW: It's always a challenge in state government. There's never enough money to do everything we want and need to do.
We hear a lot about return on security investment – how do you show your superiors that security enables business/government endeavors? Sometimes it's easier to turn that picture the other way and show the results of an incident that cost X dollars, which could have been avoided by spending X –1 dollars. That delta is oftentimes pretty significant. I'm not a big believer in FUD, but fortunately, there are always plenty of good examples of how underestimating the importance of good security controls can result in unexpected costs.
IA: And how do you get the support, resources and funding you require to do your job?
MW: Just like everyone else, I make my business case. Sometimes I win, sometimes I lose.
IA: Economy's been tight – some have experienced budget cuts, layoffs, travel freezes, hiring freezes and more. How did you fair?
MW: California state government had a budget deficit of over $25 billion last year and is facing a $20 billion deficit again this year. We've had some layoffs and also had three furlough days each month since last summer. To say it's been a challenge is a gross understatement. Fortunately, through creativity and with a frugal attitude, we've still been able to move our security program forward. That's a testament to our leadership and perceived value of the state security professionals.
IA: Do you foresee more of these stressful budgetary challenges in 2010?
MW: Yes.
IA: In regard to compliance demands, what are your priorities and how do you adhere to such regulations?
MW: State government has all of the same compliance requirements that many private sector companies have, including PCI, HIPAA and IRS regulations. Each agency and department establishes their own priorities based on their compliance requirements.
IA: If you have a number of mandates to which you must answer, how do you avoid duplicating efforts to address these?
MW: This is difficult in our decentralized state government environment because different agencies and departments get funding from a variety of sources and, in many cases, there is no elegant way of sharing either the funding or the resources. This leads to duplication of effort. My long-term goals are to offer a range of security services from a centralized security organization that agencies and departments can phase-in to their own security programs and, over time, begin eliminating the duplication of security functions.
IA: What are some of the major challenges you believe you and your counterparts at other companies/government entities face in the next year?
MW: Securing our data. I'm not the first to say it, but our traditional perimeter is gone and we need to begin dealing with that fact. We certainly still need our perimeter security appliances and tools, but data moves around in so many ways that it's almost impossible to know at all times where it's located, who has access, when it's being accessed and how it's being transferred. Our users and their mobile devices are our new perimeter, so we need to get better about data management and data loss prevention. Complexity is the enemy of security and we have some very complex environments, which can be very inviting to those who would take advantage of us. We need to decrease our exposure by decreasing the complexity.
IA: What about the major threats to your organization and its critical data?
MW: California state government doesn't really face any different threats to our organizations or data than other public or private sector organization other than the fact that government manages a vast amount of sensitive and confidential citizen and business information. So, while citizens can make their own decisions about where they shop, bank and eat out, and if they want to provide any sensitive personal information to do so, they can't make that same decision when dealing with the government. That places a higher level of responsibility in my mind for government to protect that information.
IA: What are the threats/newer applications that you think you and others in your position must address this year?
MW: We've been hearing for several years that smart phones are the new threat vector. This hasn't been a huge problem yet, but is becoming one of the things I worry about because they are so ubiquitous and difficult to manage.
I'm a huge advocate of the transparency that social media provides government, but I'm also aware of the exposure it provides. Education and awareness is the key!
IA: What's your best advice to others when it comes to building a strong security program?
MW: Recruit for talent and have good ideas. Linus Pauling said it first, “The best way to have a good idea is to have a lot of ideas.” I've found that the most talented security people will work themselves to death (if you let them) for something they believe in. These are the people you can build a strong security program around, but you've got to find them, cultivate them and give them as much creative flexibility as you can afford to keep them interested and motivated. Talented people are great force multipliers and in these times of constrained resources, we need all the help we can get.
[sidebars]
In his own words: State priorities
CSO of the Year Mark Weatherford has quite a number of objectives he'd like to meet this year as far as information security goes for the state of California. According to Weatherford, the priorities include:Enterprise security risk assessment. This is important because it will help identify where our major security issues are and allow us to prioritize funding to address our most significant problems at the agency and department level. It's going to be a huge effort.
Design and implement DNSSEC [Domain Name System security extensions] across the state enterprise. A flexible and scalable DNSSEC implementation will secure the DNS infrastructure that serves hundreds of California state agencies and local governments under the ca.gov DNS domain.
Telework and remote access security standards. As we begin making telework available to more of our workforce, it's important that we do it appropriately and securely. I sometimes hear people talk about telework like its some trivial change in organizational capability when, in fact, depending on which workers you are talking about, can open incredibly big holes into your IT infrastructure and potentially expose sensitive information.
Social media education and awareness. Social media isn't a technology problem, it's a people problem, and continuous awareness is the only way we will stay in front of it. CISOs can just put this one on their daily task from now to forever!
ON TAP: More priorities
When asked what other specific projects are on tap for this year and maybe starting in the next, Mark Weatherford replies:
Formalize our enterprise California Cyber Incident Response Team (CA-CIRT) by identifying specific talent within the various state agencies that can be called on to respond in the event of complex security incidents
We are working with the California Emergency Management Agency and the California Highway Patrol to develop an enterprise automated incident response system that will allow all state agencies to consistently and efficiently manage their security incidents.
Deploy an enterprise, web-based information security training portal to more consistently accomplish our state-wide security training requirements.
Weatherford adds that he'd also like more DAR and DAT encryption, as well as enterprise identity management.
