Predict, don’t react

Great. Well, thank you so much for joining us today on our webinar, which involves predictive maintenance. We’re really excited to have a discussion with you today about this really important topic, and I think something that’s of extreme importance to the life sciences community.

I’m joined today by three other colleagues in the Applied team. Mike Thompson, who’s the senior director in our marketing group, Amy Doucette, who’s a US sales manager, and Lucas Vann, who’s an automation engineering manager with the Biomanufacturing Training Education Center. He’s going to run us through a demo today with real equipment, actually, showing how predictive maintenance can actually be implemented on the shop floor.

So very, very excited to show you that demo today. In a very ambitious agenda, we’d like to sort of walk you through kind of why we believe the Applied story is an interesting one to the pharmaceutical community, what the key issues, challenges, and approach that we take to predictive maintenance. Walk you through a case study, Amy’s going to do that, that looks really at this important issue and what kinds of predictions we can expect in the pharmaceutical manufacturing space, and then take you through that demo.

So very, very excited, very ambitious agenda today. Just a couple of key points as we are kicking off this webinar. There are a lot of people on the call today, but we really do want to get your feedback.

So as we’re going through the webinar today, if you could just ask questions, type anything that you want to have answered into the chat window, and we will do our best to answer that as we’re going through the webinar today. So please, don’t be shy. We do love to get that feedback, and it does really guide the course of the webinar that we’ll be giving you today.

With that, I’d like to introduce Mike Thompson to you. Mike is a senior director, of course, at Applied Materials. He’s also a pioneer and thought leader in the application of simulation and scheduling technology, and he has a tremendous history and background in the semiconductor industry.

He’s going to take us through why we believe the semiconductor industry has so many lessons that they can give to the pharmaceutical community. Mike? Okay, thanks, Rick. It’s good to be with all of you this morning.Some of you may not be aware that Applied Materials is the world’s number one semiconductor and display process equipment company. It’s headquartered in Silicon Valley, generating approximately $15 billion in annual sales, employing over 20,000 employees and contractors in 17 different countries. In addition to being the leading materials engineering company, we are also the undisputed leader in factory automation software, where it is installed in 100 percent of the fully automated wafer factories in the world.

In the late 1990s to early 2000s, the semiconductor industry transitioned from semi-automated factories to full automation. This necessitated numerous advances in process technology, advanced analytics, and control and automation, and Applied was a key innovator in these areas. This slide compares key process indicators between semiconductor and pharma.

We believe that there are many similarities and opportunities to transfer software technology and lessons learned where Applied can help pharma reach new levels of manufacturing performance. Applied is at the forefront of Industry 4.0. There’s really been about four industrial revolutions, the fourth happening about now. But the first industrial revolution happened in the late 18th century, and it was enabled due to power generation, where improved productivity was realized from the introduction of steam and hydraulic power.

Then the second industrial revolution happened at the start of the 20th century, with electricity and assembly lines paving the way for mass manufacturing, improved infrastructure, and advances in financial markets. And the third industrial revolution was focused on electric automation in the 1970s to 2000s, advances in computing and the Internet for information capture and transfer in this revolution. And the current fourth revolution is happening now with digital supply networks and the execution of connected products, customers, supply chain, delivered by a vast network of cyber-physical systems.

Applied Materials Software is bringing Industry 4.0 to manufacturing plants around the world now. Back to you, Rick. Great.

Thank you so much, Mike, for that introduction. So what we wanted to do today was to sort of look at some of the key challenges and issues as we see it in the maintenance space and life sciences, and sort of look at how, you know, potential tools might be used to address those issues. So for many of you, I sort of think about the issue very simply, right? If you talk to maintenance technicians on the floor, right, there’s often a sense that when, you know, that your car breaks down, right, that we could have actually predicted that maintenance event earlier.

But actually, that there’s a lot of things that could have been done to improve our uptime of key pieces of equipment in the facility. So there’s a lot of research and a lot of information in the community today around what the perceived benefits of a predictive maintenance system, whatever that might be, right, it’s sort of trying to get advanced information about maintenance events. And we wanted to actually open a poll with you today to find out what your opinion on this important topic is.

Do you believe that, what do you believe the most important benefit of predictive maintenance is to the pharmaceutical industry today? So we’re going to launch a short poll here. And we’d be very interested to get your opinions on this. Do you believe, you know, it’s to reduce the maintenance frequency on equipment, to allow more convenient scheduling of maintenance tasks, to reduce the risk of product loss or adverse events during manufacturing, to reduce manufacturing impact, right, so the impact of maintenance events on the manufacturing space, or of reducing labor requirements for maintenance.

So we’ll just give you just a couple more seconds to respond to that. I think it’s a very interesting sort of question. Because I think the pharmaceutical space has some quite special sort of requirements in it.

So just another couple of seconds. Great. And let’s close that poll out and take a look at it.

I think the data is actually incredibly compelling here, that there’s a lot of sort of productivity improvement benefits around people’s perception of predictive maintenance. But I think for the pharmaceutical community, this idea of risk is very, very important. And you can see that actually reflected in this poll today.

69% of folks really believe that this single benefit is the most important to the manufacturing community. Okay, so why is that? Well, I think it’s very interesting that if you sort of look at the data around kind of the cost of failure, as we call it here, this is a two by two graph showing the cost of a defective batch, essentially the cost per lost batch, and the cost of a lost productive hour in the manufacturing facility. And you can see very much very clearly, aside from the semiconductor industry, I think the pharmaceutical industry has one of the highest costs of failure of any industry.

Why is that? Well, there’s a few reasons, right? Our facilities are very expensive to build. And so each hour that they’re down is a lot of money lost, not just in raw materials, but just in depreciation costs for those facilities. We make a relatively small number of batches, right? So each lost batch is a comparatively expensive proposition, especially for biomanufacturing processes, of course.

And finally, and I think, and potentially most importantly, regulation has a very dramatic impact in this industry. And it means that rework or the reworking of a batch isn’t really possible in that traditional manufacturing sense, like you see it in the clothing and textiles industry, or in the chemicals and plastics industry. So really, in a very dramatic way, and I think the poll really shows that today, keeping our equipment working and reliably producing batches is an incredibly critical thing in pharmaceutical manufacturing.

And really, the only other industry where we see that to quite the same degree is the other industry which Applied software services, and that’s the semiconductor industry. So what we’d like to do is to sort of drill down into why is it that our maintenance systems are so reactive today. So, you know, I think today for most or almost all of you on the call, our maintenance schedule is determined by the time and the frequency of use of those pieces of equipment.

Typically, it’s something that comes out of the manufacturer’s manual around how often they think maintenance should be performed. And then we use some kind of real-time equipment monitoring, some kind of basic alarming system usually, right, to catch maintenance events and breakdowns, essentially, right, where we have to go and repair equipment. But of course, there’s a tremendous amount of past data around that piece of equipment and the processes that we’re really just not using today, and that we could very much, that could very much inform the way in which maintenance could and should be performed.

I think one other issue that’s very, very important to talk about briefly is that there’s a very siloed view today in the manufacturing space, and particularly in around manufacturing equipment, right. So we have a quality group, they’re testing things, we have an automation group, they’re programming the equipment, maintenance is fixing, right. So each one of these groups has their own view into the same piece of equipment, but really, they see their own space as being very siloed.

And of course, this means that maintenance has been their job as, quote, unquote, fixing the problem, right, means that their starting point, right, is that they’re fixing problems, they’re really being inherently reactive, right. So they’re reacting to a problem, not really trying to predict when maintenance events might be needed. And so this is a real sort of challenge in this industry as we see it.

We still like to think about this problem as being kind of this analogy of having a fence at the top of the cliff, right, not an ambulance at the bottom. So what do we really mean by that? Well, you know, the ambulance at the bottom of the cliff for us is a lot of the cost associated with facility downtime, equipment repair, and lost batches, right. So there’s a tremendous cost associated with not being able to produce in today’s community.

What we want to do is to set up these fences that sit before the problem occurs, in order to be able to kind of get a more effective response. So what does this mean? It means we want to predict early, the earlier the better. We want to predict more modes of failure.

So not just kind of the most common mode, right, but of course, this being complicated equipment, there’s many modes of failure, different types of failure we need to detect separately. And finally, the idea of predicting accurately. So trying to avoid false positives, so alarming when we shouldn’t, right, or false negatives, where we don’t detect.

So this is a very, very important, I think, concept that we’re sort of missing today, when we’re thinking about maintenance, and sort of this reactive way. Okay, so let’s sort of look at kind of the broader picture. The, FDA and associated regulatory bodies, I think, have really recognized that, you know, this is an important, this idea of kind of predicting or anticipating problems before they occur is very important.

They’ve actually framed this in terms of process verification, right. But I really think that if you think about maintenance, maintenance is just, it’s just an equipment centered view, right? It’s just looking at the equipment rather than the process, right? But there’s sort of two sides of the same coin. What do I mean by that? Well, this is a CPV document, you can see here, there’s some really interesting words here, right? So we want continual assurance that the process remains in a state of control, right? So not just reactive, but continual assurance, we want to anticipate and prevent problems, right? I predict them, right, not be reactive.

And this idea here that we should be using some sort of statistical methods, right? So some sort of not just kind of a back of the envelope calculation, or kind of our own best estimates, but some kind of like statistical methods or some kind of rigor associated with collecting this information. So this is CPV, and we really view this as sort of part and parcel of the kind of the bigger picture around maintenance and manufacturing maintenance. We’d like to open up another poll now and to talk to ask you about your opinion about what we call a roadmap to predictive maintenance.

And so I think there’s a few things here. So first, on the far left hand side, you can see sort of manually performing root cause identification, right? On the right hand side, what we believe is kind of the gold standard in the semiconductor industry, which is closed loop control for maintenance. So this is where pieces of equipment within their SOP and the limits that are set for that piece of equipment are altering their performance in real time to adjust for data that they’re seeing on the floor.

So this is very much prescriptive on the right hand side, and sort of manual root cause identification on the left hand side. We’re going to open up a poll now and just get your sense of where your manufacturing organization is on this roadmap. So do you believe you are manually performing root cause identification today? Are you doing some basic predictive analysis? So like, you know, FMEA, it’s like wishbone diagrams, right? And then using that to kind of update maintenance frequency? Are you doing real time prediction? So a real time predictor uses multiple sources of data.

So quality process and fault data? Are you doing, are you building learning models? So do you have machine learning algorithms? Or are you doing any of this kind of closed loop control? Just a second more on that. And I think the results here are also very, very interesting. And so we see, just to share that the results of that poll with you, most folks, right, are using, I would say, very basic predictive analysis today.

And many of you are doing this kind of manual root cause identification. So we’re very much at the beginning of this journey, this roadmap towards predictive maintenance. And I think when we’re thinking about these kinds of solutions, we need to make sure that we have a tool and a strategy that encompasses all of these steps.

So it doesn’t just allow us to move to let’s say a level two, but allows us a full sort of path all the way up to this kind of state where we can kind of automatically issue a maintenance ticket, we can automatically reschedule. That’s where really a lot of the value is of these predictive maintenance programs. Okay, so what we’d like to do now is to talk a little bit about Applied’s approach to predicting maintenance.

And of course, we have a tremendous history of doing this in the semiconductor industry, partially because of that, that map that we looked at the beginning where, you know, it has a very high cost of failure. And so we’re very interested in protecting equipment uptime. And so the idea of predictive maintenance as Applied sees it is to build a series of shields around equipment and processes.

These use a lot of different algorithms. And we don’t want to spend a lot of time on algorithms today, but we have a ton of them. So machine learning, soft sensors, single and multivariate statistical process control, as well as, of course, regular scheduled checks.

The idea is that these sort of the shield provides a set of circles that sort of protects this process around it. And so kind of if you imagine this sort of things, trying to interrupt this piece of equipment, we detect those issues earlier, and we detect them with higher frequency. And that really allows us to kind of schedule that maintenance without that equipment being broken.

There’s a ton of… I think you’re still showing the poll here. Can you just look back to the slide? Yeah, of course. Yeah, thank you.

Thank you very much. I appreciate it. Yeah.

Some reason the poll didn’t close. Did that come up? Okay, now? Great. So hopefully, you should be able to see a diagram of that now.

This is so you can see on the left hand side is sort of a shield of around our equipment and processes, right? multivariate SPC, soft sensors and machine learning, lots of different algorithms, right? But the idea, I think, is pretty simple, right? Build a shield, a series of shields around our equipment that allows us to detect these like red balls, these problems earlier. And this, of course, has a bunch of other benefits as well that we’ve talked a little bit about. Those are shown on the right.

But I think the main sort of goal of our predictive maintenance programs should be to keep those equipment up and running as with a higher frequency as we can. Okay, so let’s look a little bit at how we do this. And Applied is the software package that allows us to model an entire workflow.

And the way I sort of think about this workflow is that it’s sort of trying to collect information from engineers and those people that have really know about how a process works, and use that information to detect anomalies early. How does it work? Okay, so we have a bunch of pre configured workflow blocks. We’ve built these over about 15 years, they capture all of the intricacies needed from PDM.

What do they do? They pull in data from lots of different systems without doing programming. So PI, LIMS systems, MES systems, right, SYNCADE, Delta V, VIRM, all sorts of other systems like that. They allow us to build a model that could be something very simple, right? A simple alarming strategy, or it could be something incredibly complex, right? A machine learning model, an expert driven model.

And this allows us to detect anomalies early. Okay. This is an important set of things to do.

But I really believe one of the kind of key benefits of having a system like this, this workflow engine is the box on the far right. And that box is critical, because it’s this idea that we want to not just detect a problem, but give people actionable data or actionable information that allows us to, for example, be alerted of the specific problem and what the most likely causes, open a ticket in the maintenance system automatically, and potentially in the future, right, it’s subject to regulatory requirements, of course, doing some kind of closed loop control, right, within the bounds of our SOPs. And so, in order to do this effectively, I think there’s sort of three key things that I’d like to focus in on and double click on today, right? So the flexible data idea, right, the idea of a predictive model.

And I think one of the most interesting trends and interesting things that we see here, you know, in the Applied spaces is that we’re kind of leveraging both machine learning, which we call sort of fully automated, you just kind of pop it into this black box, and it does a prediction with what we call expert driven modeling. So what’s expert driven modeling? Well, it’s using kind of mechanistic sort of fundamental principles, kind of analysis, you know, what’s the rate at which we would expect something to heat up? And how does this relationship between weight and temperature and time and other kinds of variables, what do we expect to see, and to try and build prediction models on that, and to do all of this without kind of buying new sensors, without buying kind of other things that we have to put into our facility. So we call these soft sensors, using our existing data to try and predict better.

And finally, on the right hand side, alert to action. So let’s drill down and look at some of these in some more detail. The idea of the workflow engine, and I think this is something that’s very, very special and unique to SmartFactory Rx, is that it integrates across the entire ISA 95 levels that you see in a manufacturing system.

So this is if we want to connect to the SCADA or DCF level, right at the bottom of the process, we can do that we can connect to LIMS and historians, we can connect to the maintenance system itself. And we’re going to show you a little demo of that today, when Lucas kind of goes through, his demo to show you how we integrate with manufacturing systems, as well as DCS systems. So very, very cool, right? Connecting into like all the way up and down the system’s hierarchy to pull data out of it and use it to kind of inform these kind of better, better decisions.

Prescriptive maintenance pulls in all of this information and allows us to kind of generate what we call a strategy. And a strategy just has a bunch of these little blocks, you kind of drag them onto the page, right? And these do things like doing analytics, notifying maintenance, generating a work order, and allows us to kind of pull in these sensors and do this kind of analysis, build the strategy in real time. So very much sort of I think about this, like, it’s kind of like a Wikipedia, right? It’s allowing putting data in here, right, that allows us to sort of a knowledge management system, right, but that knowledge management system is running on its own moving forward.

So it allows us to kind of do very, very advanced things. So let’s look at the kind of data it could pull, right? So it could be from a DCS system, it could be from maintenance systems, all sorts of CMMS systems, all sorts of different systems here. Lucas is going to be showing you this in just a second.

We can window regions of common performance. So if you think about a bioreactor, classical bioreactor, right, there’s a number of different phases of bioreactor performance or bioreactor cleaning, right, as another example of CFP or SFP operation, we might want to collect in these inside these windows, different sets of data. So this could be something very simple, a maximum value, right? It could be something a little more complicated, like a slope.

It could be something like a mean and sigma, right? So we could be looking at classical statistical process control. There’s all sorts of cool prediction models that you can build. But it starts off with this kind of idea of building a window, right, inside a recipe, that allows us to kind of look at common patterns of behavior that we’d expect to see inside each one of those windows.

The other thing that this tool allows us to do that I think is very cool, and we’re going to demo to you today, is not just to alert somebody, and many of you are on email chains or get text messages that have alerts on them, right? The idea here is that we don’t just alert you, but we provide actionable data. What does that really mean? So you can see here an example of an email. It shows you, tells you what the outlier it is.

It tells you what the last maintenance event was. It links the last maintenance report, right? It analyzes to find what the most likely cause of failure is. And it allows you a simple button that you can go click to see more data, right? This is very valuable because it decreases the amount of time that it takes a maintenance crew to react to an event when they see it, okay? And this is, I think, the really critical thing to avoid kind of losing batches, right? That we want to make sure that we do it as far as possible, okay? There’s lots of different things that we can do with this, opening tickets, integrating with a production schedule, or even closing the loop.

And I think, you know, many of you want to do just a small number of these. We see kind of over the life cycle of a manufacturing facility, people doing more and more sophisticated things around this kind of alert to action. But you can start really simple if you want to and sort of build over time.

Just to sort of close the loop on this kind of SmartFactory Rx product, it’s a very sophisticated offering. It’s been used for more than a decade in the semiconductor community. It has a tremendous number of libraries of sensors that are based on kind of support vector machines, partially squares, multiple linear aggression, proprietary algorithms, lots of different stuff.

You can also integrate it with a lot of other packages. We know that many of you today are using lots of advanced analytics packages, and it integrates directly with those too.

If you’re using MATLAB or R or some kind of third party package we can integrate with that, and I think that’s a very, very nice kind of thing to do. The idea here is to provide actionable information, right? So to provide a wrapper around those analytics packages that allows us to actually act more quickly. Finally, what we’re going to do today when we show you the demo is to show you two sets of user interfaces.

So, Lucas is going to walk you through how we engineer a workflow today. So, this is this idea of pulling in data, configuring actions, defining relationships. This is something we don’t do a lot, so we might do it weekly or monthly.

We do this on an installed app, it’s tremendously powerful and it’s enterprise-enabled. On the right-hand side, you’ll see we also have a kind of a web-based solution. And we believe this is really important for operators and for people on the manufacturing floor.

Also, for management, right? To be able to click with a simple button and see a dashboard, a set of indicators, right? Like a gauge, right? A set of information that’s very simple, that’s sort of bulletproof, and tries to summarize a lot of this information. So, just something to keep in mind. We have both of these user interfaces set up and they’re both part of the same package.

So, what I’d like to do now is to transition over to Amy. She’s going to walk through with us today a case study where we looked at trying to predict maintenance events early, right? So, to try and move that fence as far away from the edge of the cliff as we can. Amy?

Thank you, Rick. So both Mike and Rick touched on how we’ve been able to fine-tune our solution for predictive maintenance in the semiconductor industry, but we’ve also had a chance over the past couple of years to prove our solution’s value in enabling predictive maintenance for pharma. For this use case that we’re sharing with you today, the customer had many different types of unplanned maintenance, but their most critical unplanned downtime was occurring on their bioreactor temperature control units and agitators.

We were able to demonstrate with both univariate and multivariate models that we were able to predict over 70 percent of events, six days in advance. Being able to predict that far in advance really enables you to reduce the risk of lost products. So by connecting their assets into our platform, applying predictive analytics, integrating maintenance scheduling, we were really able to provide enhanced asset management.

And however many unplanned downtime events you have a year that impact manufacturing of your product, that is the manufacturing cost and revenue that you are able to capture back with SmartFactory Rx. To dive a bit deeper into our overall approach to implementing predictive maintenance, once we integrated the data into our platform, we needed to understand which runs were good and which runs were bad on the equipment so that we could model the variability. As we captured more data, we were continuously tuning those models and applying them to monitor the equipment in real time for predictive and prescriptive notification.

This becomes really important as not all like-for-like equipment behaves the same, and we also have the ability to set different parameters dependent on which product you are running and apply it to the model. So this particular equipment was used in a multi-product facility, and in order to handle it, we’re showing you just one way here how each different product may have its own respective limits. And we have the ability to handle different process parameters depending on which product you’re running at that moment.

Once your limits are understood and applied to your models, our drag-and-drop strategy editor, which we’re showing down below, allows you to take automated action by integrating directly into a CMMS system, sending a report, and notifying an employee to action with prescriptive notification in less than one minute. So the value we’re able to prove in a short period of time in this use case really had extremely impressive results. It allows the manufacturing support team to proactively focus maintenance efforts ahead of any problems and drastically reduce unplanned downtime.

What we’re going to do now is move into our live demo to show you some more of the functionality within our platform. Lucas Vann is one of our application engineering experts in both biomanufacturing processes and equipment, and he’s going to show us how he has Applied SmartFactory RX at BTEC at North Carolina State University. Great.

And what we’re going to do now is just switch over to Lucas. Lucas, do you want to share your screen for us? Yeah, sure. Perfect.

Great. And Lucas is actually showing here the actual BTEC facility, which is why we’re switching over to his computer today. Yeah, good point, Rick.

Thanks. This is a live facility, and we’re going to be running an actual sterilization cycle on an actual bioreactor. As Rick mentioned earlier, I’m the automation engineering manager here, and I wanted to first begin with a failure that we had some time ago where we actually lost a production batch.

Now, as a manager, I do get received emails and texts that come from our DCS system, and this is an example of one that really has two issues. One, in terms of the content, oftentimes it’s a little cryptic, and it’s very difficult to understand what’s going on, but also these are usually coming in the most inopportune of times, in the middle of the night or on weekends, when there’s limited resources available at the facility. And so, the first thing that I do normally when I get an email like this is that I have to then research and get into our DCS system, and this is an overview of our facility.

We have utilities. We have our upstream, our downstream, and also our support utilities in terms of the support vessels of media or buffer prep. And if it’s a reactor that was identified in the email or text, I have to then navigate to that vessel, look at the vessel, and if I want to see the trends of how things were changing on the way to this point where I got this text, I can open that up within the DCS.

Some of the issues that we have, though, is that it’s sometimes difficult to view the problem, and I’m going to scroll through to a batch that we had, and you can see a little bit about the nature of some of these trends. Difficult to interpret, difficult to see color variation sometimes, and there’s scaling issues in terms of being able to find out the exact value. So, this, even if we were able to detect, and oftentimes the TLK, the weight was increasing at a certain amount, I want to know what kind of maintenance has been done.

And again, this is a separate system, another system that then I have to log into. And so, we use a maintenance connection, which is maybe a CMMS system that some of you are familiar with. And so, when we log in, I need to then, again, navigate to the piece of equipment at hand.

We have a number of different assets here at the facility, and I’ll bring up the actual vessel, so you can see some of the most recent work orders that are coming in. But if I want to know a particular on that vessel, I have to navigate to the area that it’s in and then to the vessel itself, and then it will pull up the actual list of history for the different work orders that were done on this particular system. And so, the most recent one was a mechanical PM, and in that mechanical PM, I know I can drill in further and find that, yes, there was a mechanical seal that were changed.

And mechanical seal failure oftentimes results in abnormal amount of condensate bleed through and then volume change or increase in my vessel, and I end up with a lost batch. So, what happened here, in terms of the alert, was that we already had lost the batch. And in our case, we were looking at ability then instead to be able to pursue a better alternative.

And the initial thought was to increase the frequency of some of our preventative maintenances. And this actually resulted in more failures. And what we really wanted to do was have the ability to monitor equipment health and do more of this predictive maintenance.

But without having the advanced sensors, we wanted to use process variables. So, we don’t have a lot of, we don’t have vibration sensors or ultrasound or acoustic. And even those, as Rick was mentioning earlier, really need process data for context.

So, there’s still the ability or the need to bring everything together. So, our solution was to reach out to another industry, to the Applied Materials, and look at their type of installation. I want to show you now the type of email that we get from the similar instance here, where we have a predictive model set up.

So, this is before the failure actually would happen. And again, it gives us some time to correct the problem. So, in this email, Rick was mentioning the context that can be added.

So, even in the title here, I have a configuration for which batch it was, which reactor. The alert is identifying that I have a sort of a holistic model here. This is the seal health monitor multivariate model that’s telling you the exact date and also what occurred.

But with here, there’s some prescriptive action. So, even if it’s at an inopportune time, when an inexperienced engineer is on site, they have this right here at their fingertips to say, this is where you need to go and look. Almost like an expert engineer would have this type of knowledge.

At the same time, we have a link to the dashboard, which gives us even more information. And so, I don’t have to start delving through and looking at different sort of parameters or my DCS system and pulling up analog trends. Everything is right here.

So, this is the dashboard. This is the model that we’re running. And I can easily click on any one of these runs and see the contributors, things that are contributing to this particular batch being outside of limits.

So, you can see we have limits here. And these are configured based on just what our statistical normal operation would be. And so, here we see that one of the main ones that is causing a problem is this seal health for this point being outside of limits.

We see there are some other contributors too. That the heat up was different. That the weight range was different than most of the other batches.

That the pressurization, the slope or increase of pressure was also different. And so, we want to then delve deeper into this. And we also noticed in those emails, we have a univariate trend as well.

And so, this is then showing each of these batches as we progress through the runs throughout the facility. And seeing how there’s actually a trend here in terms of this custom equation of seal health. Even though we don’t see, you look at the trends on the right here, that each one is going to be highlighted in red.

We see very little difference in terms of how the reactor is heating up. The actual temperature achieved is still the sterilization temperature and it cools down. But there is a difference when you start looking at more of a soft sensor or combining multiple types of sensors together. And so, where this is actually used or how we drive these dashboards is going to be first through a strategy. And so, I want to bring you into some of the strategies that we have. This is the one that is running for both the univariate type of model that you just saw on the dashboard and the multivariate.

And so, we see here there’s a number of things that the model is running. There’s actually a command here, this application, which is being used to query our CMMS system. So, again, we’re within this strategy being able to pull data from disparate systems like our CMMS.

So, it’s going to issue a series of commands to log in and pull that data and then be able to attach that to this report using this audit block. And so, again, you can see, as Rick mentioned, a number of different blocks available. We could use Python, a Python block or a MATLAB block, and now we’re in real time able to run this code to make it useful, make it applicable.

And then at the end, we have this notify block where we send the email. And this is also configured here where we can pull up different variables and put them in. It’s going to be linked to data coming from this actual model.

So, these can be simple or they can be quite complex. If we look at, this is a bit more of a complex strategy that’s running. And we have a number of different things here.

One is based on a particular concentration decrease in the batch as our batches are running. So, we have equipment health models. We also have process health models.

And this is telling me that there’s some abnormal consumption related to low B vitamins. So, if I see too low of a slope, this is related to a particular metabolite difference within that batch. And we usually, when we see that, we have low protein yield.

So, we want to be known or notified of this. We also have cases here that are saying when parameters, variables that are coming from, again, different sources, we have PAT analyzers, we have turbidity probes. These are coming from different sources, but we’re pulling them together here in this strategy to then make an action on it.

So, when they reach these values, I want to issue a command to actually turn a pump on. And this is that command going back in and driving into that to turn that pump on. So, we have a number of different things.

And this last one, a bit more complex, is monitoring for condensate buildup in our exhaust filters. And if those filters begin to get clogged, and we’ve seen that these models will predict this hours earlier before they become clogged, we can start making a change to allow that filter to recover. And so, these are some important things that we’re looking at when it comes to our… Yeah.

Lucas, just a quick question that came in. Can you tell us a little bit about, from you… You’re obviously an external user of the system, not like an Applied employee. Can you tell us a little bit about your level of programming sophistication, your backgrounds, you know, how difficult is this to program and set up? Right.

It’s a very good question. So, starting off, I have no programming experience, really, other than Fortran. I’m not sure if anyone even recognizes that, but way back when I was doing undergrad.

So, my programming skills are limited. This kind of setup here of just this drag and drop, and then being able to configure each one of these blocks, it’s just sort of a short learning curve to be able to say that, okay, what are the things that are coming into this block, and how do I pass them through? But it’s just linking them up in the configuration here, in this input map for each of these blocks. So, each block is going to have sort of this input map.

And then, when I bring in a start block, I have all these things available to me to pass forward through when I’m looking at a case block. So, it’s a very simple way of just aligning up and saying, well, I want to have the equipment name being passed through so that that’s going to allow the univariate model to know which equipment it’s running. And then, when I run the model, then I’m going to pass through, and I want to have in this case block that if I get any outliers or critical or warning, I just configure it here, and that if anything happens, I want to pass that information through.

So, it’s quite a simple sort of workflow that’s easy to manage. So, it’s a little bit of learning curve at the beginning, but especially for a non-programmer, not too difficult. And one thing I’ll say is that if you are a programmer, then this becomes extremely powerful.

I’ve seen some programmers that take a look at this, and they understand the logic involved, and it’s very easy for them to come up with some very advanced solutions here. Great. Thanks so much.

Sure. The last thing that I’ll do then is I want to show you guys sort of a live reactor. And I’m going to go back to our DCS system and look at this 1250 reactor.

This is running a sterilization cycle, and this reactor is going to be running a specific model. And before actually I allow this to continue, what I’ll do is I’ll show you a bit of the modeling just so you have an understanding of where these blocks come from. If we look at our univariate modeling, this is of a sterilization cycle, and so that’s what we’re going to be running on that reactor.

The important thing, as Rick mentioned earlier, is that we have different phases or different areas of importance. And as an engineer, I know that there are specific things I want to look for in each of these windows. And so, we can use process conditions to set these windows.

If we look here, we have four different windows. Two during the heat phase, because I know there’s an operation that occurs that closes the valve, and then pressure begins to rise. So, my heat up and different parameters will change in this window.

As an engineer and a process person, I understand that, and I can build that into the model. Once we have the windowing, then we’re very easily able to look at a number of different statistics. The software comes configured with a wide range of statistics, and also the capability to build our own statistics, in a sense.

And so, that’s what these custom equations are for. So, each custom equation that we have here, we have the ability then to actually write sort of equations that can be more mechanistic in nature. For temperature control, we understand that there’s a certain force or an output from the temperature controller, and we should see a response from the jacket.

And we can look at that relationship and maybe make an equation based on that relationship. Or we can go straight back to heat transfer or thermodynamics. That’s something that we’ve done here, whether it’s from a window basis, or in this case, this heat transfer equation.

This is actually what’s called the data transformation. So, every collection event, when it’s coming in, I’ve been putting an actual heat transfer equation that’s looking at the rate of heat transfer from the jacket to the vessel. And if this is going to be changing, I can monitor how that changes or if it’s going to change abnormally by now looking at statistics around the heat transfer during that heat update.

So, very configurable, very easy to bring batches in to the model. So, we have a number of runs here that you see that have been brought in. We can create context because oftentimes we have a lot of different runs.

Well, I can choose any one of these different context variables. Now, I’m adding context. So, when I bring these batches in, I know that I’m only looking at sterilization batches and ones that have been effective.

So, very easy to create that context, very easy to then retrieve new runs coming in based on that context. And then, in addition to that, then to start modeling. And so, what you end up seeing are more of these statistical process control charts here, where if I’m interested in the slope of the jacket, I see how that variation is occurring from run to run across the batch.

And then, also, the idea of that potential now of using a custom equation based on a mechanistic principle to see how the seal health is performing. From there, we oftentimes can bring this in. We don’t maybe want to look at just one statistic.

We want to look at more of a multivariate view. And so, we now, what we create are what’s called hybrid models because we have these mechanistic relationships that we know are important, but we’re now putting that into more of an empirical model. So, we have now a hybrid of mechanistic expert knowledge being built into this empirical data set.

So, we look at the same batch that was showing an issue in the univariate stat. This is the one that’s also showing an issue in the multivariate. And again, related now, looking at more than just the seal health equation, but other parameters.

So, we again have a holistic view using mechanistic understanding, which I think gives us the most robust type of prediction model. So, using these models, we’re running a very simple sterilization cycle. And I’m going to acknowledge this manual message to allow this to start.

So, now, we’re going to see that the recipe itself is a sterilization recipe. It’s a full vessel SIP. It’s gone green.

It’s starting to run. The set of unit operations that it’s running through, you can see here that it’s in this manual message, which is knowledge, and it should pass forward into the heat phase. And when it does, you can look and see that the reactor is heating.

It’s red. That means the jacket is heating. What is actually causing the heating is this loop that we see here.

So, we have a pump that’s circulating fluid through the jacket, and we have a valve opening to send plant steam to this heat exchanger to heat the jacket. Now, what I’ve done is to simulate a valve heat exchanger. I’ve throttled back the input steam, utility steam, to this valve.

So, while it’s heating up… To be clear, this is actually on a real piece of equipment, right? Right, Lucas? Yeah. Yeah. This is actually running a reactor right now.

So, it was a manual throttling of the valve that I had to do. And this is heating up. So, it would reach temperature, take a little longer.

We wouldn’t see an actual alarm in our DCS system. And that’s one of the things that as an engineer, I want to know that the health or that the performance is deviating slightly. And so, that’s what we’re able to see here is that the performance potentially will be deviating from what we would hope to see.

And we can look at that in a number of ways. There’s a debugger here, which allows us to see this model is being executed. So, every single time we started it, while it was in this waiting period, nothing is running.

It’s not reaching the limits of this particular batch. When we do see that the reactor will fire a model or something happens, we should start seeing that this notify block would go. And so, that… Here, I’ll show you the…

The actual block here, which is quite interesting because we can follow with this debugger. This is just a sort of monitoring. It’s saying that it went through and it found that there was a calculation done for the UVA, but met the statistics.

There was an outlier. And so it reached us notify block and it actually sent this notification through. So I’ve set this notify block, the link to just a Gmail address that I’ve set up for you guys to be able to see instead of my own personal one. And when we go to that, this email, I should see here in my inbox that I have a new email that has come through and it’s on that bioreactor at 1250. There’s the heat up excursion that was detected. And what kind of information do we have there? Well, the severity of that is a level two.

I’ve also done a calculation based on what the target should be and what the actual result was. So it gives me a percentage of effectiveness in a sense of that heat exchanger. So it’s deviated down to 18%.

And normally what we’d want to see is sort of a, maybe a slight deviation. It’s 100%. It’s right on target.

And then maybe it’s slowly deviating down. On top of that, we went out to the maintenance connection system and we were able to pull in the latest maintenance report work order that came through on that vessel. So now I can start linking up to see, you know, what was, was there something done maybe to that heat exchanger? Why it’s performing differently? And if that’s the case, sometimes we have the ability then to say, well, there’s new limits of operation because of this, this maintenance that was done.

Maybe this is a different operation, but maybe it’s okay, right, from a maintenance perspective. And so all this information together really helps us to determine that and then decide whether or not moving forward, we need to have sort of limits being generated based on this new normal shift of operation. And with that, I think that I can turn it back over.

I think one other thing just to mention, we do have other systems here at BTEC and this is our SCADA system. Now, while it’s with alarm and alert, we have no way in previously of getting that out. This is a proprietary system that we didn’t have a link to email.

Now, with this no change in our SCADA system, we actually have the ability to run the models on this reactor and actually not only get alerts, but with that same level of information associated with it on reactors that are not even part of our VTS system. So really functional, whatever the type of vessel it is on whatever type of system, we really have, it’s increased our capability here quite substantially. Great.

Thank you so much, Lucas, for that very comprehensive and really, I think, really cool walkthrough. What I’d like to do now is to hand back to Amy. I know a lot of you are starting to ask questions in the chat window of Lucas, and we do want to answer as many of those as we can.

But we would like to firstly just sort of tie this back to the bigger picture of what Applied Science is trying to do. Amy? Thanks, Rick. And thank you, Lucas.

And before we, as we transition into the next piece, I just wanted to add on to Lucas’s demo that we do realize not everyone is checking their email in real time. So it was the easiest way to demo today. But we also do have mobile text messaging capabilities.

So that if you’re out on the manufacturing floor, you may not be checking your email, but you would get that alert through mobile phone. So Applied Materials, we have built a team of pharma experts with experience and reliability, and we really have a deep understanding of both process and equipment health analytics for pharma. We understand that the road to Smart maintenance can be very overwhelming, but we’re ready to help you achieve success every step of the way, from building models to applying them into automated strategies with the help of our application engineering team. With our platform, what we’re showing on the right-hand side, you have the potential to achieve integrated Smart maintenance. So not only advanced reporting and visualization around equipment health, but also dynamic maintenance scheduling, prescriptive corrective action, and as far advanced as automated corrective action. Applied Predictive Maintenance Solution, Rick, if you want to go.

Thank you. Is part of an integrated IIoT platform where we connect shop floor advanced data analytics to the equipment asset management system, and both the manufacturing and maintenance schedules. So we’re not only focused on driving out variability within a single process, but across your entire business for end-to-end real-time precision manufacturing.

We have the ability to make industry 4.0 a reality for pharma with our wide library of algorithms that we’re showing here on this slide. This is really going to support pharma’s move from reaction to prediction and prescription. And we’ve been delivering this innovation successfully for the past 30 years, the semiconductor industry.

We truly believe that pharma could really learn something from SEMI as we move towards the industry, the vision of industry 4.0, and we want to work together with you to build resilience into your equipment and processes. We’ve already proven successful results in pharma, and we look forward to providing you with the opportunity to transform the way you manufacture high quality products for your patients today. Great.

Thank you so much, Amy. I really appreciate that. So there’s a ton of questions just coming in, and I wanted to kind of kick things off by asking, we’ll answer just as many of those as we can.

The first one really to Lucas around kind of this mechanistic modeling kind of principle. Could you say a little bit more about kind of the types of quote unquote mechanisms, right, that you found useful in kind of doing this modeling? You know, so I think you mentioned sort of heat check that kind of this sort of temperature sort of basis, but what other kinds of things are there? So I guess the first thing to also mention is that mechanistic often is more related to first principles, right? And so by mechanistic here, it’s more fundamental understanding of operation. So it’s not necessarily first principles, but it’s still a mechanistic sort of functionality that as an engineer, as a person who runs the equipment or the process, we understand that this relationship should hold true.

In some cases, it can be first principle, like heat transfer or thermodynamics, but it oftentimes we’re building models in the case of that seal health. We knew that some content they build up in the reactor during a sterilization cycle is normal, and we were looking for abnormal. But that abnormal increase can also be if the pressure, the steam pressure regulator is set too high.

So we had to take in multiple sort of parameters, and we can look at the relationship between those, that the difference between the vessel temperature and the jacket temperature in relation to the overall increase in weight and start inferring that, hey, there’s something different here based more on the fundamental understanding of how this should work with the parameters that we’re looking at. So whether it’s a temperature controller, knowing that for this amount of output, the jacket temperature should be changing more, or a relationship that’s a bit more complex, that pulls in maybe four or five different parameters to build really a soft sensor, in a sense. So it’s a soft sensor that using the statistics that are critical for more of a fundamental operation.

Yeah, and I think one of our attendees has actually provided another great example, compaction equations, right, being included in a roller compaction for a tablet press machine, right? So that’s another example of these quote-unquote custom equations that use some sort of set of physics principles or other knowledge that we have around the facility. One thing, you know… Right, even drying, granulation drying. Exactly, right.

We’ve looked at that. Yeah. Yeah.

And that’s helpful, right? Because I think, you know, one of the kind of key differentiators of the pharmaceutical industry from many other industries, right, we don’t have a tremendous batch history, right? We’re not like a consumer packaged goods company is making millions of batches a year. So there’s a tremendous wealth of past batches in terms of the count of those batches that we can use to kind of drive machine learning and other things. We think in pharma, this idea of these mechanistic or physics-based, right, or custom equation principles are really, really important because they kind of get that, they allow our, you know, engineers’ detailed knowledge of a facility to be incorporated into a model.

So I think that’s a really cool, kind of one of the really cool and very unique features about this tool. Can you say a little bit about how the BTEC organization as a whole has responded to, you know, this inline control? Because I know a lot of the folks on the call today, you know, are very aware that they’re in a very regulated environment, right? They have a set of SOPs and limits that are part of their filing. And so, of course, there’s a concern that, you know, if you do some sort of closed loop control, that might not be something that’s part of the regulation.

Can you say anything about that, Lucas? Yeah. So we do have the advantage that we’re a pilot-scale facility, and we’re not too concerned. We can actually trial these new, either new sensors or new softwares, and also feedback in and control the process.

But oftentimes, in view of that, we can often win the model and notify an operator, right? So in the case of our fire reactor exhaust filters clogging, one of the options, instead of feeding back in and opening up or changing parameters, it would be to notify an operator that this is about to happen, and within limits of operation, because validation always occurs, as everyone knows, within limits for certain parameters, we can affect changes and suggest those changes so that an operator then would be able to make that change in that validated state. Yeah. So sort of a semi-automatic, right? So have the operator still in the middle of that loop, but in a very much more kind of, in a way that alerts… Informed way.

Exactly, exactly. Yeah, yeah. Great.

Good. Just one more question. We’re very short on time today, but there’s a question here, I think for Amy, around kind of the software offering that Applied has.

Can you say a little bit, Amy, about… I know that we focused today on very much the software. Can you say a little bit about, for people that… Obviously, this is a huge and overwhelming set of data for folks. How can people kind of get started working with Applied? How do our service offerings sort of tie into this? So the screen that we’re on right now gives my contact information.

Please feel free to reach out after this meeting. It is a very wide platform of different solutions. So right now we have four, analytics and control, which is focusing on real-time monitoring and control around both process and equipment health.

We have advanced maintenance, which is the CMMS system. And the integration between the CMMS and analytics and control is really going to help you enable predictive and prescriptive maintenance. Operations productivity.

We had a previous webinar with more information around this solution that we can share with you. And this is everything from planning at the high ERP level to modeling a facility before it’s even built, all the way down to real-time scheduling and dispatching at the manufacturing floor. And last but not least, we’ve, in over the last few months, just moved over a fourth solution from the semi-industry around knowledge management, because we’ve heard that there’s such a large gap in pharma around this.

And we are taking this OCAP, ECAP process and making sure that we’re making it fit well for pharma and improving upon it with customer feedback. So it’s very easy to, what we would say to you is take a look at your facility and what is some low-hanging fruit? What are some of the high-value problems that you have that we could easily come in, help you solve, improve the value of bringing in a platform like this? So get a foot in the door with one of the solutions, and then we can take a look at the platform as a whole. So please feel free to reach out to myself or Mike Thompson.