As we navigate a more connected, digital landscape, patch management is now a more complex process with exponentially larger implications to your business’ profitability and reputation. On July 26th, 2018, the Department of Homeland Security issued a warning regarding a spike in ERP system attacks. ERP systems contain a trove of personal data, a common target for nation-state hackers and other cybercriminals. Two investigative reports were issued on July 23rd, 2018 by security firms Onapsis and Digital Shadows on this activity spike.
The reports identified detailed information regarding SAP hacking and even SAP HANA-specific exploits being exchanged in Russian-speaking forums on the dark web. According to the researchers, many of these attacks are using previously identified vulnerabilities for which fixes have been released by the ERP vendors but not yet applied on certain self-hosted ERP systems. The reports also state that hackers go after cloud-based ERPs lacking established, strong security measures. These cybercriminals will even leverage leaked username and passwords on the dark web from other security breaches in previous years.
At the point these reports were issued, there were over 4,000 security patches for vulnerabilities in SAP applications and more than 5,000 for Oracle. In fact, researchers found about 50 exploits for SAP products and another 30 for Oracle that are being discussed on the dark web.
This warning highlights the importance of patch management for ERP systems to ensure that at least identified vulnerabilities are remediated. Additional procedures for strong passwords, proactive penetration monitoring, and a repeatable process will help reduce, but not eliminate, the security exposure of a company’s ERP data.
Patch Management in the ERP Landscape
Many companies that have implemented ERP systems like SAP or Oracle over the years have focused on the point of deployment with little regard and planning for periodic patching of the systems. ERP systems are extremely complex due to the shear amount of functionality, complexity of architecture, variability of enterprise data, millions of lines of code, etc. It’s not surprising that the software companies provide bug fixes both individually and bundled as releases. At the same time companies are paying for this service in the annual license maintenance fees (typically 22% of license costs) paid to the ERP companies.
Some of these patches fix functionality whereas some are released as legal updates on an annual basis based on changes in law from different countries. Unlike cloud environments, on-premise deployments must apply patches manually. In cloud environments, the software manufacturer is typically applying patches in releases multi-times in a year to the environment. In those situations, customers will have their system updated automatically. The ERP software companies provide a schedule for those releases, so companies can test and prepare to ensure their functionality is still working as releases are applied. For on-premise deployments, it is up to each customer to plan, test, and ultimately choose to apply patches on a period basis.
Though it may seem simple, the process for applying patches on a periodic schedule can be arduous. Usually if a specific software bug is encountered, an individual note is released by the software manufacturer, requiring a focused testing effort to ensure the functionality is fixed without harm to other operations.
However, when these notes are pulled together as a bundled release (Support Stack in SAP terminology, and Bundle Release Notes for Oracle), there is a tremendous amount of complexity to ensure that fixes are appropriately applied and existing functionality within a company’s ERP environment is still operational, particularly custom functionality. This often requires a series of regression tests on existing end-to-end functionality and even creation of specific targeted tests for key fixes that are known to be within the bundles.
An industry best practice for applying released bundles on an ERP landscape includes a periodic process, usually semi-annually or annually. A set series of key regression tests which represent the core processes within a company can be leveraged. These core processes are usually integration scenarios across workstreams like order-to-cash, procure-to-pay, plan-to-produce, financial processes, etc. These would also include not just internal ERP transactions but integration to 3rd party systems as needed. Targeted testing for a key process may also be necessary depending on the number of notes.
Many companies have not planned for a periodic process and can be left years behind on applying notes. In many of those situations, new legal requirements or changes to business practices may necessitate new functionality, which can be slowed as the system must come to a minimum patch level to actualize the new functionality. In these circumstances, the effort to apply the patches could be more time-consuming and cost more than an overall upgrade or new implementation.
Patch Management in Regulated Industries
In some industries (i.e., life sciences), federal regulations can draw additional scrutiny from entities like the FDA. As an example, within FDA-regulated industries, computer systems must adhere to a specific set of processes when building and releasing software (i.e., CFR 21 Part 11). These validated processes ensure that the system is built according to a set of user requirements, which has robust documentation, stringent testing, and traceability through the process from requirements through testing results and deviation remediation.
This rigor can add that additional layer of complexity for companies in applying release bundles. Having said that, that is a perception more than a reality if a company’s computer systems validation protocols follow a risk-based approach as allowed by the FDA. Many life science companies do have periodic programs for applying release bundles on a regular basis.
Regardless of the system or software your business is utilizing, strong and sustainable patch management principles are critical to protecting your business and your customers. With cyberthreats looming, a well-managed process for testing and applying patch updates could be the very thing that ultimately ensures business continuity.