Implementing NLP in industrial process modeling: Addressing categorical variables

• Published in: Computers & chemical engineering Vol. 199; p. 109146
• Main Authors: Koronaki, Eleni D., Loachamín-Suntaxi, Geremy, Papavasileiou, Paris, Giovanis, Dimitrios G., Kathrein, Martin, Czettl, Christoph, Boudouvis, Andreas G., Bordas, Stéphane P.A.
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.08.2025
• Subjects: Categorical variables; Feature importance; Natural Language Processing; Regression models; Shapley analysis; Feature importance; Regression models; Shapley analysis; Categorical variables; Natural Language Processing
• ISSN: 0098-1354
• DOI: 10.1016/j.compchemeng.2025.109146

Important variables of processes are often categorical, i.e. names or labels representing, e.g. categories of inputs, or types of reactors or a sequence of steps. In this work, we use Natural Language Processing Models to derive embeddings of such inputs that represent their actual meaning, or reflect the “distances” between categories, i.e. how similar or dissimilar they are. This is a marked difference from the current standard practice of using binary, or one-hot encoding to replace categorical variables with sequences of ones and zeros. Combined with dimensionality reduction techniques, either linear such as Principal Component Analysis, or nonlinear such as Uniform Manifold Approximation and Projection, the proposed approach leads to a meaningful, low-dimensional feature space. The significance of obtaining meaningful embeddings is illustrated in the context of an industrial coating process for cutting tools that includes both numerical and categorical inputs. In this industrial process, subject matter expertise suggests that the categorical inputs are critical for determining the final outcome but this cannot be taken into account with the current state-of-the-art. The proposed approach enables feature importance which is a marked improvement compared to the current state-of-the-art in the encoding of categorical variables. The proposed approach is not limited to the case-study presented here and is suitable for applications with similar mix of categorical and numerical critical inputs. •Inputs represented by categorical values are embedded by language models.•Dimensionality reduction is used to reduce size of resulting inputs.•The similarity between categories is realistically represented.•Accurate predictive models are trained using new input embeddings.•Realistic feature importance is enabled using Shapley analysis.