A practical framework for turning weak, individually unreliable signals into a single predictive susceptibility score, using bottom-side internal corrosion as the worked example.
What do you do when a failure mode is real, well-documented, and still invisible to your inspection program? Wait, or build around the blind spot. Internal corrosion, and specifically bottom-side pitting, is the clearest example of this bind. It's been the documented cause behind roughly 12% of hazardous liquid and gas transmission pipeline incidents over the past decade, per PHMSA, and the defects that cause the worst surprises tend to sit right at the edge of what your ILI tool can confidently detect and size.
One solution is to wait for better inline inspection tools… Another solution is to harness data science to combine several weak signals into one strong one, and the process for doing that is more generalizable than it looks. Here’s the framework, using bottom-side internal corrosion as the worked example.
The instinct is to reach for a complete physics- or chemistry-based model of the threat. For internal corrosion, published probabilistic models exist that incorporate 60-plus parameters: fluid composition, microbiological activity, operating conditions. The problem is that almost no pipeline operator has that data at the granularity those models need, for every mile of every pipeline.
So don’t build that model. Build a phenomenological one instead: a model that doesn’t try to explain why the threat behaves the way it does, only to recognize the pattern it leaves in data you already have. For most operators, that means ILI anomaly data and elevation profiles. If your starting question is “what do I already have, reliably, on every segment?” you’ll end up with a much more usable model than if you start from “what would a complete theory require?”
This is the part that takes judgment. You’re looking for factors that are each, on their own, a mediocre predictor, but that tend to show up together near known failures. For bottom-side internal corrosion, five factors fit that description:
None of these alone would have flagged a release that was the impetus for this susceptibility modeling project approach2: a 17%-depth pinhole sitting just outside the bottom-side orientation window, missed by the prior ILI run. The susceptibility score comes from multiplying the factors together, so a region only scores high when several weak signals line up at once, which is a much rarer, much more meaningful event than any single factor crossing a threshold.
What would the equivalent five signals look like for a threat you’re trying to model, like dents, SCC, or geohazard exposure? The factors will differ, but the principle holds: look for what consistently shows up together near known failures, not what theory says should matter.
Once you have candidate factors, you need a way to know if the combination actually works, and “it looks reasonable” isn’t a validation method. Build a dataset of confirmed incidents (PHMSA’s public records are a starting point for corrosion-related releases), filter for cases with available data at the failure location, and use that as ground truth.
From there, run a grid search across your adjustable parameters (window size, outlier sensitivity, factor weights) and score each combination by how highly it ranks the true failure regions relative to every other candidate region on the same line. For this model, the best configuration achieved an average ranking performance of 0.82 across the validation dataset, with a perfect score on two of the pipelines. The tuning process matters as much as the selection of factors: a good set of signals with poorly chosen weights will still underperform.
After tuning on your dataset, test the model against a case it has never seen. We applied the tuned model to a pipeline with two confirmed pipeline failures in the form of a release that weren’t part of the original dataset. The first pass caught one of the two releases. After folding that pipeline into the dataset and retuning, it caught both. A model that only works on the data it was built from hasn’t been validated. It’s been memorized.
Nothing about this approach is specific to internal corrosion. The same four steps apply just as well to other threats where a single inspection reading falls short: work with data you have, find individually weak signals that cluster near known failures, validate against confirmed incidents with a proper ranking metric, and test blind on holdout cases. If you’re staring at a threat on your own system that no single dataset seems to predict reliably, it might not be a data problem. It might just be waiting for someone to combine the data you already have in the right way.
If you want to see this same philosophy pushed to full rigor, our team's paper at this year's IPC does exactly that for SCC susceptibility. Using ILI and soil data from six pipeline operators across more than 30,000 km of pipe, we built a simple weighted classifier structurally identical to the score above, then compared it against a fully trained XGBoost model, including a hard look at how the wrong validation split can quietly inflate your results. Catch the full paper, "SCC Susceptibility Modeling to Demonstrate New Guidelines for Machine Learning in Pipeline Integrity," at IPC 2026 in Calgary.
1. Determining Active vs Passive Internal Corrosion Using Data Science - Yevgeniy Petrov, Megan Scudder. PPIM 2023
2. Identification of regions highly susceptible to release due to bottom-side internal corrosion - Yevgeniy Petrov, Megan Scudder. AMPP 2025
Looking for conference papers? Check out our Resources section.