The Many Roles of a Security Group in a Technology Company

August 26th, 2016 Permalink

Security in technology is an increasingly specialized business these days, but there is a lot more to working in a security group at a software company than just the matter of vulnerabilities in code released or under construction. This is a challenge, as most companies, technology companies or otherwise, invest too few resources into the security function. Sadly, as a rule lack of investment is true for most things that are additional costs and look a lot like an insurance policy: spending on security to reduce the risk and scope of vulnerabilities typically shows little in the way of obvious returns in the short-term, especially if the people involved are good at what they do. Large, horrible events like intruders setting up bitcoin mining operations in your cloud accounts, or stealing your mailing lists, or publishing your password databases don't happen with any great frequency in the comparatively few organizations that any one individual works with over the years. Rarity breeds complacency. When in charge of the purse strings, there is always the pressure to cut the costs that are not clearly and directly linked to business metrics, hoping that lightning strikes elsewhere for the rest of your tenure. A great deal of effective leadership in security is a matter of success in fighting that pressure, and ensuring that sufficient resources are available to achieve all of the necessary goals.

Compliance and Due Diligence

Assisting Customer Due Diligence

It is an unfortunate truth that many of the well-established tasks relating to security are arguably of little practical value, but a company nonetheless does have to carry out these tasks. Most cannot be avoided. For example, consider the responses that must be provided to shallow security due diligence questionnaires arriving from potential and existing customers. Assembling these responses is a manual, time-consuming, and unfortunately necessary part of dealing with large companies - and is much worse for government bodies. For the most part vendor assessment at this level is little better than security theater. In the worst case two sets of uncaring individuals far removed from any topic under discussion exchange documents that neither understand, have little reflection of actual realities, and that will never be referenced again. In the more likely case, one can do the thankless work to provide an accurate response, and it will still receive little material attention. The only thing that the typical exchange of documents and efforts proves is that a company is willing to spend money to check this box - there is a lot more signaling taking place than the creation of any actual value. For a mid-sized company with large customers or many customers, carrying out this function can easily chew up all of the human resources available for security.

More reasonable activities include assisting a customer in running network and application scans against the products they will be using, or carrying out meaningful audits of underlying technologies and practices. A presently small but growing number of larger companies undertake this sort of assessment of their vendors: while it is welcome, it does require a lot more effort on the part of the security team and other parts of the organization. For companies with many customers, if even a tenth of them decide to take this path that can represent a significant workload over and above the more usual exchange of information about security practices.

There is a very real risk to acknowledge here, which is that company leadership will look at this large slice of the work of a security group, the most visible part of security work from their perspective, and see little reason to do more. This is one of the few parts of security work that is directly connected to revenue, in the sense that it needs to happen for deals to be signed or renewed, and is arguably the least important in any other way. Therefore it will always be prioritized. Given that much of this process is nothing more than theater, there is a lot that can be done to reduce costs without compromising ethics, however. For a reasonably stable product, it is possible to produce templates, checklists, and processes that provide sufficient guidance for junior employees to perform a majority of the work, to pick one example.

Assisting Regulatory Compliance

There are a number of areas of regulatory compliance usually handled at least in part by a security group. For example, PCI compliance for credit card payment systems involves dealing with a third party scanner and certification organization, and tracking or carrying out any necessary work needed to remediate issues. Personally identifying information (PII) has to be cataloged and managed, requiring continual attention as products grow and change. The security group must be involved in the design and architecture process of all new software and services with regulatory compliance in mind, and will have to collaborate with the company legal team in order to interpret often vague law into concrete implementation details.

Assisting with Vendor Due Diligence

The products provided by vendors must be assessed from a security standpoint: are they in fact secure; what is the additional cost of deployment and ongoing management in order to maintain sufficient security standards; how can they integrate with company authentication services; and so forth. In this day and age most companies use vendors for critical systems in which important data is stored by the vendor, not by the company. It is important to understand the implications and obtain representations that the vendor is managing the data responsibly. Some of this is driven by regulation and other legal requirements, but attempting to minimize real risk is a very important part of this process. It is entirely appropriate and reasonable to, for example, arrange to run network and application vulnerability scans against vendor systems.

Defensive Preparations

Physical Security of Buildings and Assets

Workplaces require physical security. Everything from locks to guards to vendors for badge systems may wind up being managed as much by the security group as by the operations group within the company. Certainly the security team should at the very least be advising on best practices and the process of auditing outcomes.

Security of Employee Machines

Employee machines require defenses against theft and malware, as well as effective backups for company data. This is the case whether issued by the company or owned by the employees themselves, and whether the machines in question are desktop, laptop, or mobile devices. Those defenses require setup and ongoing administration, and employees require support for security-related technical issues such as attacks via browsers and email, application and OS updates, encryption of data, use of the company network, and so forth. This infrastructure and support is typically managed by the IT group within a company, but the security group assists by providing and maintaining specific guidelines for the systems and software used by the company, and delivering prompt notice of new vulnerabilities that require action.

Network Security

The broad bucket of network operations for any given company might include setup and management of VPNs, office networks and associated physical equipment, directory systems such as LDAP for centralized authentication, third party content delivery networks (CDNs), cloud infrastructure, response to DOS and DDOS attacks, and much more. On that last point, it is not a trivial undertaking to adequately defend against DOS and DDOS attacks, and most organizations do not have the necessary knowledge to create that infrastructure. These days third party CDNs and cloud providers offer services to mitigate attacks, and the security group should help to assess vendors and ensure that these services are set up and used appropriately. The same goes for all other aspects of network infrastructure: the network operations group will be largely responsible for the work and planning related to security concerns, but the security group guides, advises, and audits.

Internal Accounts and Permissions

The company directories used for login, authentication, and access rights are an important concern. Ensuring that access rights are appropriately managed is a very necessary part of organizational security. As organizations grow, as more people, internal, and third party systems are added, it becomes ever more important to get this right. A proliferation of directories and login systems is to be avoided, as it makes it hard to grant and revoke access rights in a timely fashion, and hard to audit the current state of the system. The desired goal is a master central directory, with a good user interface and ability to implement best practice password management rules, two-factor authentication for all users, single sign on integration with all third party applications, and rapid change of user access permissions according to circumstances. The utility of two-factor authentication when it comes to preventing breaches really can't be overemphasized - it is one of the best additions to access management when judged by cost and benefit, and should be instituted earlier rather than later.

Both third party and internally developed applications require ongoing management and auditing of access rights, usage, and related configuration. Are user accounts removed when employees leave the company; do sessions expire after an appropriately short period of time; is stored data secure; does the application expose any unnecessary functionality; are access rights appropriate for users, granting only the least required privileges; is the application appropriately monitored; and so forth. The security group usually guides rather than runs this process for any specific application, but must absolutely be on top of ensuring that every application adheres to best practices. When any of these items fall short, it makes it that much easier for attackers to find their way into company systems, and that much harder to find and close the holes.

Application Design and Architecture

The security team can and should influence the development of existing and new software products. Instituting and running security reviews as a part of the process for the development and deployment of a new application is a good approach. Early reviews are better than later reviews, as it is easier to change course if necessary. The first review can take place at the design stage, or during initial development, and a later review prior to final deployment to check on the details. Later reviews can be scheduled as needed when large changes are forthcoming, but within a development organization that has adopted a security review process and run with it for a while, it typically becomes the responsibility of development groups to reach out to the security organization to organize reviews as needed.

Securing Data

Taking steps to minimize the chances of data loss or theft is a part of application design, and an important concern when considering the management of user access and privileges, but also encompasses the protection of company documents, communications, and information stored locally, in company machines, or by third party vendors. Regulatory requirements in most jurisdictions require much of this data to remain intact. Backup strategies must be established and audited for all company data, and this may include a very wide range of devices and systems, maintained by a mix of in-house and vendors. Further, all data should be ideally be appropriately encrypted both at rest and in transit, wherever it might be. In some cases that will be a regulatory requirement, but in this day and age wall-to-wall encryption of everything is both a viable goal and a good idea.

Detection and Repair of Vulnerabilties

Announced Vulnerabilities

All companies must patch and update libraries and systems on a regular basis as vulnerabilities are discovered and announced by the broader community. All too few do this as well as they might. The role of a security group here is twofold; firstly to ensure that new vulnerabilities are noticed by those who need to know, and secondly to ensure that the appropriate systems are updated in a timely fashion. The first point is actually an interesting problem, as to be effective it requires a catalog of systems and libraries, accountable contact points for all development and operations groups, and system of management that works well with high priority interrupts.

It is surprisingly hard to do this well, even for dangerous and widely announced vulnerabilities such as those in OpenSSL from recent years. It is very easy to flood people with too many announcements, for one. Without a good system of coordination and work management, and without a willingness to respond on the part of developers, the security group might well find itself tracking down a remnant set of vulnerable servers a year or more after announcement of a critical vulnerability. One approach that can make a big difference is to set up a central clearing house application that reads vulnerability announcement feeds. Teams subscribe to the clearing house for the specific applications, libraries, and versions they work with, and can also mark specific vulnerabilities as being mitigated or irrelevant to their present situation. A team will only see the meaningful announcements, and the state of subscriptions and updates can be used to determine where the security group needs to follow up.

Centralized Vulnerability Detection

One of the more visible activities carried out by a security group is the use of scanning tools that browse company networks, APIs, and web applications in search of potential issues. Typically this is very visible because it involves advance notice and the subsequent delivery of voluminous reports that must be processed and understood at length, usually filled with false positives, and then a flurry of tasks relating to discovered vulnerabilities that must be completed. The setup and management of these tools is very time-consuming in most cases, and the reports produced tend to require considerable effort and some specialized knowledge to digest into a useful set of information that can be acted upon by the average developer.

This isn't the only way to run this sort of process, and finding something better should be a priority, but almost every potential solution requires the expenditure of lot of time from more experienced security team members, or developers with a fair amount of knowledge of security issues. Scanning reports certainly shouldn't be dropped raw on the teams that will have to act upon the results, however. Since the process of interpretation tends to require both knowledge of the specific application and knowledge of security and the scanning tool in question, some form of process or infrastructure is needed to ensure that both sides educate one another and that scanning can quickly be made a cost-effective activity that proceeds without a lot of additional effort after the first few times. Piping the results into some form of intermediary clearing house application, in which false positives can be flagged and teams need only pay attention to the issues affecting their work directly, is just as useful in this case as for other areas of vulnerability management.

Some tools in the vulnerability detection toolkit can be distributed to development teams, run by the teams for their own specific applications in the same QA processes that identify functional bugs. Where possible this is always desirable, as it spreads knowledge and awareness of security concerns within the development organization. Static analysis tools are a good candidate for this sort of distributed usage: at minimum identifying out of date and vulnerable libraries, but many can do more than this, and the best can identify some of the same classes of issue as can be found by scanning a running application. Static analysis isn't a replacement for scanning - both should be carried out on a regular basis - but it is is much more amenable for the average development group to set up and use. A security group can teach and advocate, and then leave the actual work to the developers in this case.

Security Assessment and Penetration Testing

Engaging an outside group to carry out penetration testing and assessment of security practices is a cost-effective way of uncovering important issues that would otherwise pass unseen - the different perspective that outside security experts bring with them is a valuable part of the service. At the same time, the outcome of a short exercise in penetration testing will, by virtue of where it finds vulnerabilities, help to identify those parts of the company that need more attention from a security group. It is almost always the case that specific applications or services fall into the cracks between organizational structures, and in consequence fail to receive the attention they merit. Thus it is not only helpful but arguably necessary to carry out third party assessment and penetration testing on a fairly regular basis, and it is the role of the security group to identify vendors, manage the process, and ensure that value is both obtained and acted upon.

Intrusion Detection

The systems and processes that allow detection of intrusion must be determined, created, and maintained. A security group may or may not be responsible for running these processes and the accompanying infrastructure applications, but should certainly be driving the intrusion detection strategy. The approaches uses might include real-time monitoring of cloud infrastructure usage, audits of access attempts to all systems of interest, triggers for unusual usage of credentials, and a layer of attention paid to a subset of the usual devops monitors and alerts, those most likely to fire in the event of activity on the part of a hypothetical intruder.

Incident Response, and Managing and Tracking Remediation

It isn't enough to find vulnerabilities and notify the relevant groups responsible for development within the company. The vulnerabilities must also be addressed, possibly urgently if they were discovered as the result of active intrusion or exploitation. Incidents of this nature will always arise at the worst possible time, and a process must be in place to ensure that the response proceeds without problems and delays.

In most organizations, new vulnerabilities disrupt the development roadmap, and thus it is very important to have both an established, agreed-upon remediation process and a commitment from leadership that normal work can and will be delayed where necessary to fix or obviate new vulnerabilities. An important part of any such process is a transparent internally published metric for success: show which teams are succeeding and which are not. Given that vulnerabilities can be managed in much the same way as bugs and development tasks, a metric should emerge in exactly the same way as it does for other tasks: a vulnerability can be graded by severity and its remediation prioritized in work queues in the same way as any other job. All companies above a certain size will have task management systems and a culture of task management, and tying the security process into both system and culture is an important part of the buy-in from company leadership when it comes to addressing vulnerabilities.

If this process and cultural integration is not well established, then security issues will be addressed at best in a haphazard fashion, and at worst not at all. Existing development roadmaps will always be prioritized by default. Integration of security with management of development at the detail level is one of the most important tasks to address up front when arriving in a growing organization that has not yet formalized its security processes. Leaving it to good faith efforts just doesn't work, sad to say.

Security Training

For Non-Technical Employees

A required part of all employee training is general security awareness covering topics such as phishing and other dangers, good practice in use and storage of credentials, and so forth. Setting up training is usually a matter of working with one of the many third parties who put together training packages, and then ensuring that everyone passes through the training program. There will probably be aspects of regulatory compliance or best practice as determined by the legal team involved as well, which inform the contents of the training materials, and the frequency of training. Much of this often falls under the umbrella of the human resources group, and so it may be the case that a security group only needs to source and vet the security training materials to ensure that they are suitable.

It is perfectly possibly to build your own security awareness training packages, using the OWASP materials as a starting point, for example, but why reinvent the wheel? It is much less costly in time and resources to use any of the established products in the marketplace, and life as a security specialist is as much about doing more with less as is it is about security, unfortunately.

For Software Engineers

Security in software development is a complicated business, but that isn't any excuse not to become familiar with the basics, meaning the OWASP top ten list or similar materials. Avoiding the most common and easily created vulnerabilities should be something that junior developers understand at the level of writing good code, and something that senior developers further understand at the level of instituting good checks and incentives in the development process. If you can talk about the best way to set up a development pipeline and testing for functional correctness, you should also be able to talk about how to minimize the creation of vulnerabilities and test for them.

Unfortunately teaching the basics in any formal way is still pretty rare in the profession, there are far too many software engineers who don't really understand enough of the security space in their part of the field, and there are all too few good pre-packaged resources out there for someone who wants to build a training program. The OWASP materials are great as a reference to point to for people who have a grasp of the outline already, but need a further level of digestion and organization in order to be useful in training programs. So a security group should expect to have to spend time to assemble and understand a good training package tailored to the types of product and branches of software engineering present at the company. It can't be completely hands-off: going team by team to run through OWASP presentations, question and answer sessions, and collaborative attempts at one of the few prepackaged capture the flag exercises is a good basis for this sort of training, for example.

Teaching the Security Mindset

More than just the specifics, it is important to work towards a culture that considers security in the same way that it considers other good work practices - that security is on the list of requirements for any project, and is in mind when people are asked about how they do their jobs. The knowledge required to consider security aspects of a project should spread by virtue of the fact that ever more of the participants understand the basics and understand the cost-benefit considerations of security in the workplace, and particularly in software development. Achieving this goal requires sustained advocacy for security as a necessary goal, and a security group with the time and resources to carry out outreach into the organization beyond the obvious touch points of training sessions and showing up when there are problems.

Contributing to the Risk Management Picture

The minutiae of risk management account for a large part of the business of running a company. In order to usefully allocate resources and determine strategy, a clear picture of risks, opportunities, and costs is required. The security group must take all of the varied threats and activities outlined here and digest them for presentation to company management: what needs to be accomplished, what must be accomplished, the severity of particular threats, the cost of mitigations, and so forth. Keeping that digested view up to date and useful is a necessary part of the bigger picture of risk and opportunity.

Standards, Policies, and Documentation

All of the myriad items mentioned above tend to involve complicated tasks of many steps and numerous decisions that require discussion. To prevent slowing of work and continual rehashing of issues, members of the security team must in the course of their work develop and publish a set of standards, policies, and supporting documentation for both themselves and the rest of the company. This library will range from how-to recipes and checklists for complicated and infrequent tasks, to standards for functions common to many applications such as authentication or encryption of data, to policies that will become a part of the employee handbook, to training materials, and much more. This can be a real time sink, but avoiding this necessary task will cost even more time in the end.

In Summary: there is a Lot of Work Here

It should not have escaped your attention that this is a long list, many parts of which are only sketched here at a very high level. The activities of the security group that are naturally most visible to upper management make up only a fraction of everything that needs to be accomplished, which is one of the reasons why security is often underfunded as a function within companies. The work needed to make a company and its products secure can and should be distributed out beyond the security team, spreading the knowledge, motivation, and will to take action out into the broader company community. That in and of itself requires a significant investment in process and persuasion. It all starts with explaining how this whole business of security must work to achieve results, and then proceeds by gaining understanding and agreement with that vision at all levels of the company. Without that, and within the budget granted by that understanding, it is hard to get far.