Could Reinforcement Learning Play a Part in the Future of Wafer Fab Scheduling? [Tech Paper Review]
A discipline of Machine Learning called Reinforcement Learning has received much attention recently as a novel way to design system controllers or to solve optimization problems. Today, Jannik Post – one of our optimization engineers – takes a look at the background of the methodology, before reviewing two recent publications which apply Reinforcement Learning to scheduling problems.
The exciting prospect of Reinforcement Learning
Traditionally, semiconductor fabs have relied on real-time dispatching systems to provide their operators with the dispatch decisions – with their ability to show the current state of the work in progress within seconds. These systems may follow rules based on heuristics or derive them from domain knowledge, which makes their design a lengthy process that requires deep knowledge of the fab processes. Maintenance of the contained logic also requires continuous attention from subject matter experts. As well as this, these systems have very limited awareness of the global effects of decisions at toolset level – therefore making them susceptible to providing suboptimal decisions.
More advanced approaches to wafer fab scheduling rely on optimization models, which can take many factors into account, e.g., the effect of dispatching decisions on bottleneck tools further downstream. These solutions will generally require a slightly longer computation time to achieve high-quality solutions.
Reinforcement Learning (RL) promises to avoid the downsides of both common dispatching systems and optimization approaches. So, how does it work? At the heart of RL there is an agent* which performs a task by taking decisions or controlling a system. The goal is to teach this agent to make close to optimal decisions by allowing it to explore different options and providing feedback on the quality of its decision. Good decisions are rewarded whilst suboptimal decisions are punished. Of course, this training will not be performed in a live environment, but rather by simulating thousands of scenarios that might occur to prepare the agent for any possible situation.
A common example of Reinforcement Learning is self-driving cars, but it can easily be seen how it could be productive when used in other environments, such as dispatching in a wafer fab. In theory, it could be utilised to dispatch wafers to tools in a way that optimizes certain KPIs – such as throughput.
Reinforcement Learning for Job Shop Scheduling** problems
Numerous recent publications have explored the use of RL for production control. However, the approaches are still in their early stages and applied to problems much less complex than semiconductor scheduling. Nevertheless, they demonstrate the potential to play a part in future solution strategies. Two approaches stood out to us when reviewing the literature:
“Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning” (2020)
This paper by Zhang et al. describes an approach to designing an agent that generalises its knowledge beyond what it has been trained to do, enabling it to handle unseen problem instances. This is achieved by initially conducting a large amount of diverse training scenarios. The model can flexibly handle instances of different sizes, e.g., with varying numbers of tools.
The agent is first trained on large numbers of scenarios and will thereby learn to exploit common patterns and perform well in instances not encountered before. After the training, the agent can be deployed to solve new instances. As training is conducted separately from solving an instance, the latter can be performed in less than a minute. The performance on benchmarking problems is compared against optimization models and simple dispatching heuristics. The Reinforcement Learning approach yields a makespan – the total duration of the schedule from start to finish – between 10-30% longer than when computed through optimization, but around 30% shorter than what simple heuristics achieve.
“A Reinforcement Learning Environment for Job Shop Scheduling” (2021)
This paper published by Tassel et al. sets out to design a reinforcement learning environment to optimize job shop scheduling (JSS) problems as an alternative to optimization models. The objective in this approach is to reduce periods in the schedule where tools are not in use, which is shown to correlate with a minimisation of makespan. The agent is designed as a dispatcher and is trained on a single scenario at a time by running a real world simulation over and over. As the goal is to generate an optimized solution for the instance, the best solution achieved during training will be saved. Training time and solution time are thus the same in this approach and are limited to 10 minutes to reflect production requirements. In this approach, there is no intention to generalise the behaviour of the agent to other instances.
The authors disclose a makespan of just 10-15% worse than the best known benchmarks for job shop scheduling, and just 6-7% longer than time-constrained optimization approaches.
Flexciton’s view
At Flexciton, we are excited about bringing cutting-edge optimized scheduling to wafer fabs worldwide. We are always exploring new ways that could help us improve the service we provide our customers so it’s exciting to see new emerging technologies which may help solve scheduling challenges in the semiconductor industry. The two publications reviewed in this article both present promising new approaches that yield measurable improvements over simple dispatching heuristics, but still fall short of optimization.
Both approaches can cope with disruption and stochasticity of the environment, such as machine downtimes. Another commonality is that both can readily be applied to problems of different sizes. In both cases the authors respected the requirement for frequent schedule updates (Tassel et al.) and quick decision support (Zhang et al.) and still achieved optimized solutions. It is conceivable that reinforcement learning has the capability to teach an agent to make smart decisions in the present that will improve the future fab state and reduce bottlenecks.
However, as the use of RL for JSS problems is still a novelty, it is not yet at the level of sophistication that the semiconductor industry would require. So far, the approaches can handle standard small problem scenarios but cannot handle flexible problems or batching decisions. Many constraints need to be obeyed in wafer fabs (e.g., timelinks and reticle availability) and it is not easily guaranteed that the agent will adhere to them. The objective set for the agent must be defined ahead of training, which means that any change made afterwards will require a repeat of training before new decisions can be obtained. This is less problematic for solving the instance proposed by Tassel et al., although their approach relies on a specifically modelled reward function which would not easily adapt to changing objectives.
Lastly, machine learning approaches can lead to situations where the decisions taken by the agent will be hidden in a black box. When the insights into the rationale behind decisions are limited, troubleshooting becomes difficult and trust into the solution is hard to establish.
Flexciton’s way
Using wafer fab scheduling to meet KPIs such as increased throughput and reduced cycle time is a challenge that requires a flexible, quick, and robust solution. We have developed advanced mathematical hybrid optimization technology that combines the capabilities of optimization models with the quickness of simple dispatching systems. When needed, the objective parameters and constraints can be adjusted without the need to rewrite or redesign extensive parts of the solution. It can therefore easily be adapted to optimize bottleneck toolsets, a whole fab or even multiple fabs.
Flexciton’s scheduling software produces an optimized schedule every five minutes and easily integrates with existing dispatching systems. The intuitive interface enables users to investigate decisions in a wider context, which helps during troubleshooting and increases trust in the dispatching decisions.
References
[1] Zhang, Song, Cao, Zhang, Tan, Xu (2020). “Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning.”
[2] Tassel, Gebser, Schekotihin (2021). “A Reinforcement Learning Environment for Job-Shop Scheduling.”
[3] Five reasons why your wafer fab should be using hybrid optimization scheduling (Flexciton Blog)
Notes
* – We use the term ‘agent’ to describe a piece of software that will make decisions and/or take actions in its environment to achieve a given goal
** – The job shop is a common scheduling problem in which multiple jobs are processed on several machines. Each job consists of a sequence of tasks, which must be performed in a given order, and each task must be processed on a specific machine.
Explore more articles
View all
Staying ahead in smart manufacturing technologies has become paramount for global competitiveness. This topic was the focal point of the recent panel discussion webinar hosted by Flexciton.
The semiconductor industry's journey toward fully autonomous manufacturing is underway, driven by advanced technologies and strategic investment. Staying ahead in smart manufacturing technologies has become paramount for global competitiveness. This topic was the focal point of the recent panel discussion webinar, hosted by Jamie Potter, Flexction CEO & Cofounder. The panel featured industry leaders representing fabs and suppliers: Matthew Johnson, VP of Wafer Fab Operations at Seagate; Patrick Sorenson, Industrial Engineer at Microchip Technology; Francisco Lobo, CEO of Critical Manufacturing; and Madhav Kidambi, Technical Marketing Director at Applied Materials.
Survey Insights: Where Are We Now?
The panel discussion was initiated with a presentation of the findings from Flexciton's inaugural Front End Manufacturing Insights survey, conducted among fabs in the US, Europe, and Asia. Key takeaways included:
- A majority of respondents see autonomous manufacturing as achievable within the next decade.
- Data standardization and integration remain major barriers, delaying scalable solutions.
- Cloud computing, IoT and Mathematical Optimization stand as the top three advanced technologies that fabs have adopted so far.
These insights laid a strong foundation for a lively discussion, highlighting the shared vision while addressing divergent strategies and challenges.
Insights from Industry Experts
Pragmatism Over Perfection in Data Models
Francisco Lobo emphasized the importance of starting with what’s available when building scalable solutions.
“Instead of building a complete model from scratch, leverage existing standards and your MES infrastructure. Begin with a pragmatic approach and evolve as you learn.”
This iterative strategy ensures companies can start deriving value early, without waiting years for a perfect model to be developed.
Strategic Investments In Downturns
While many fabs postpone investments during downcycles, Matthew Johnson emphasizes that smart manufacturing investments should be continuous rather than cyclical. He highlighted the strategic advantage of such approach:
“In down cycles, you often need these solutions the most. For example, using smart manufacturing to scale metrology tools through sampling can significantly stretch your existing resources without capital-heavy investments.”
His insight underscores how downturns provide a window to refine processes for long-term gains.
Getting Leadership Buy-in
Securing leadership support for smart manufacturing investments remains challenging when benefits aren't immediately apparent. Patrick Sorenson shares that the ROI justification was easier during the recent upcycle:
"If we just get a few more lots out of the fab when we have more demand than capacity, that will pay for itself."
In other scenarios, focus on demonstrating benefits through yield improvements, capital avoidance, or labor efficiency.
Industry Alignment on the Vision
Madhav Kidambi observed a growing consensus around the end goal of autonomous manufacturing, even as companies differ in their pathways:
“The vision of Lights Out manufacturing is clear, but strategies are evolving as companies learn how to justify and sequence investments to sustain the journey.”
Ecosystem Collaboration and The Path Towards Autonomy
A key theme emerging from the discussion is the importance of collaboration between suppliers and fabs. This includes:
- Open platforms and integration capabilities
- Standardized data protocols
- Partner ecosystems for specialized solutions
- Shared innovation initiatives
As the industry progresses toward autonomous manufacturing, success will depend on:
- Maintaining continuous investment in smart technologies
- Taking pragmatic approaches to data integration
- Developing clear ROI frameworks
- Fostering collaboration across the ecosystem
- Building upon existing systems and standards
As Matt from Seagate concludes,
"Fab operation is really a journey of continuous improvement, and the pursuit of smart technologies is a fundamental tenet of our strategy to ensure that we meet the objectives as an organization."
Watch the Full Webinar
The conversation is packed with actionable insights on overcoming barriers, achieving quick wins, and navigating the complexities of smart manufacturing adoption. Don’t miss out—click here to watch the full discussion recording.
Innovate UK, part of UK Research and Innovation, has invested in Flexciton and Seagate Technology's production planning project to help improve UK semiconductor manufacturing.
London, UK – 1 Oct – Flexciton, a UK-based software company at the forefront of autonomous semiconductor manufacturing solutions, is excited to announce investment from Innovate UK in a strategic collaboration with Seagate Technology’s Northern Ireland facility. Innovate UK, the UK’s innovation agency, drives productivity and economic growth by supporting businesses to develop and realize the potential of new ideas. As part of their £11.5 million investment across 16 pioneering projects, this collaboration will help develop and demonstrate cutting-edge technology to boost semiconductor manufacturing efficiency and enhance the UK’s role in the global semiconductor supply chain.
Jamie Potter, CEO and Cofounder of Flexciton, commented:
"We are thrilled to partner with Seagate Technology to bring yet another Flexciton innovation to market. By combining our autonomous scheduling system with Flex Planner, we are enhancing productivity in semiconductor wafer facilities and driving greater adoption of autonomous manufacturing."
The partnership aligns directly with the UK government’s National Semiconductor Strategy, which seeks to secure the UK’s position as a key player in the global semiconductor industry. Flexciton’s contribution to this strategy is not just a testament to its cutting-edge technology but also highlights the company’s role in reinforcing supply chain resilience and scaling up manufacturing capabilities within the UK.
Flex Planner: A breakthrough solution for chip manufacturing
At the heart of this project is Flex Planner, the first closed-loop production planning solution for semiconductor manufacturing with the ability to control the flow of WIP in a fab over the next 2-4 weeks, autonomously avoiding dynamic bottlenecks, reducing cycle times, and improving on-time delivery performance.
Supporting the UK's semiconductor growth
The UK government’s investment in semiconductor innovation underlines its commitment to fostering cutting-edge solutions that bolster the sector’s growth. The semiconductor industry is projected to grow from £10 billion to £17 billion by 2030, with initiatives like this collaboration driving the innovation necessary to achieve these goals.
Flexciton’s partnership with Seagate exemplifies how collaboration between technology innovators and manufacturers can lead to transformative advances in the industry. The funding from Innovate UK enables both companies to develop and test solutions that not only enhance productivity but also position the UK as a critical link in the global semiconductor ecosystem.
About Flexciton
Flexciton is pioneering autonomous technology for production scheduling and planning in semiconductor manufacturing. Leveraging advanced AI and optimization technology, we tackle the increasing complexity of chipmaking processes. By simplifying and streamlining wafer fabrication with our next-generation solutions, we enable semiconductor fabs to significantly enhance efficiency, boost productivity, and reduce costs. Empowering manufacturers with unmatched precision and agility, Flexciton is revolutionizing wafer fabrication to meet the demands of modern semiconductor production.
For media inquiries, please contact: media@flexciton.com
The semiconductor industry is set to receive $1tn in investment over the next six years, driven by AI and advanced technologies, with over 100 new wafer fabs expected. However, labor shortages continue to pose a challenge, pushing the need for autonomous wafer fabs to ensure continued growth.
Over the next 6 years, the semiconductor industry is set to receive around $1tn in investment. The opportunities for growth – driven by the rapid rise of AI, autonomous and electric vehicles, and high-performance computing – are enormous. To support this anticipated growth, over 100 new wafer fabs are expected to emerge worldwide in the coming years (Ajit Manocha, SEMI 2024).
However, a significant challenge looms: labor. In the US, one-third of semiconductor workers are now aged 55 or older. Younger generations are increasingly drawn to giants like Google, Apple and Meta for their exciting technological innovation and brand prestige, making it difficult for semiconductor employers to compete. In recent years, the likelihood of employees leaving their jobs in the semiconductor sector has risen by 13% (McKinsey, 2024).
To operate these new fabs effectively, the industry must find a solution. The Autonomous Wafer Fab, a self-optimizing facility with minimal human intervention and seamless production, is looking increasingly likely to be the solution chipmakers need. This vision, long held by the industry, now needs to be accelerated due to current labor pressures.
Thankfully, rapid advancements in artificial intelligence (AI) and Internet of Things (IoT) mean that the Autonomous Wafer Fab is no longer a distant dream but an attainable goal. In this blog, we will explore what an Autonomous Wafer Fab will look like, how we can achieve this milestone, the expected outcomes, and the timeline for reaching this transformative state.
What will an Autonomous Wafer Fab look like?
Imagine a wafer fab where the entire production process is seamlessly interconnected and self-regulating, free to make decisions on its own. In this autonomous environment, advanced algorithms, IoT, AI and optimization technologies work in harmony to optimize every aspect of the manufacturing process. From daily manufacturing decisions to product quality control and fault prediction, every step is meticulously coordinated without the need for human intervention.
Key features of an Autonomous Wafer Fab:
Intelligent Scheduling and Planning: The heart of the autonomous fab lies in its scheduling and planning capabilities. By leveraging advancements such as Autonomous Scheduling Technology (AST), the fab has the power to exhaustively evaluate billions of potential scenarios and guarantee the optimal course for production. This ensures that all constraints and variables are considered, leading to superior outcomes in terms of throughput, cycle time, and on-time delivery.
Real-Time Adaptability: An autonomous fab is equipped with sensors and IoT devices that continuously monitor the production environment. These devices can feed real-time data into the scheduling system, allowing it to dynamically adjust schedules and production plans in response to any changes or disruptions.
Digital Twin: Digital Twin technology mirrors real-time operations through storing masses of data from sensors and IoT devices. This standardized data schema allows for rapid introduction of new technologies and better scalability. Moreover, by simulating production processes, it helps to model possible scenarios – such as KPI adjustments – within the specific constraints of the fab.
Predictive maintenance: Predictive maintenance systems will anticipate equipment failures before they occur, reducing downtime and extending the lifespan of critical machinery. This proactive approach ensures that the fab operates at peak efficiency with minimal interruptions. Robotics will carry out the physical maintenance tasks identified by these systems, and when human intervention is necessary, remote maintenance capabilities will allow technicians to diagnose and address issues without being on-site.
The Control Room: In an autonomous fab, decision-making is driven by data and algorithms. The interconnected system can balance trade-offs between competing objectives, such as maximizing throughput while minimizing cycle time, with unparalleled precision. That said, critical decisions such as overall fab objectives may still be left to humans in the “control room”, who could be on the fab site or 9000 km away…
How can we get there?
Achieving the vision of an Autonomous Wafer Fab requires a multi-faceted approach that integrates technological innovation, strategic investments, and a cultural shift towards embracing automation. Here are the key steps to pave the way:
A Robust Roadmap: All fabs within an organization need to have a common vision. Key milestones need to be laid out to help navigate each fab through the transition with clear actions at each stage. SEMI’s smart manufacturing roadmap offers an insight into what this could look like.
Investing in Novel Technologies: The pivotal step towards autonomy is investing in the latest technologies, including AI, machine learning, AST, and IoT. These technologies form the backbone of the autonomous fab, enabling intelligent planning and scheduling, real-time monitoring, and adaptive control.
Data Integration and Analytics: A crucial aspect of autonomy is the seamless integration of data from various sources within the fab. By harnessing big data analytics, fabs can not only gain deep insights into their operations, but they will have the correct data in place to support autonomous systems further down the line.
Developing Skilled Workforce: While the goal is to minimize human intervention, the semiconductor industry will still require skilled professionals who can manage and maintain advanced systems. Investing in workforce training and development to fill the current void is essential to ensure a smooth transition.
Collaborative Ecosystem: Even the biggest of chipmakers is unlikely to reach the autonomous fab all on their own. Collaboration with technology providers, research institutions, and industry partners will be key. Sharing knowledge and best practices can accelerate the development and deployment of autonomous solutions.
Pilot Programs and Gradual Implementation: Transitioning to an autonomous fab should be approached incrementally. Starting with pilot programs to test and refine technologies in a controlled environment will help identify challenges and demonstrate the benefits. Gradual implementation allows for continuous improvement and adaptation.
How will fabs benefit?
The transition to an Autonomous Wafer Fab promises a multitude of benefits that will revolutionize semiconductor manufacturing:
Enhanced Efficiency: By optimizing production schedules and processes, autonomous fabs will achieve higher throughput and better resource utilization. This translates to increased production capacity and reduced operational costs.
Better Quality: Advanced process control and real-time adaptability ensure consistent product quality, minimizing defects and rework. This leads to higher yields and greater customer satisfaction.
Reduced Downtime: Predictive maintenance and automated decision-making reduce equipment failures and production interruptions. This results in higher uptime and more reliable operations.
Improved Flexibility: Autonomous fabs can quickly adapt to changing market demands and production requirements. This flexibility enables manufacturers to respond rapidly to customer needs and stay competitive in a dynamic industry.
Cost Savings: The efficiencies gained from autonomous operations lead to significant cost savings. Reduced labor intensity, lower material waste, and optimized energy consumption contribute to a more cost-effective production process.
Sounds great, but when will it become a reality?
The journey towards an Autonomous Wafer Fab is well underway, but the timeline for full realization varies depending on several factors, including technological advancements, industry adoption, and investment levels. However, significant progress is expected within the next decade.
Short-Term (1-3 Years):
- Implementation of pilot programs and continual adoption of AI, IoT, AST and other advanced technologies.
- Incremental improvements in scheduling, process control, and maintenance practices.
Medium-Term (3-7 Years):
- Broader adoption of autonomous solutions across the industry.
- Enhanced data integration and analytics capabilities.
- Development of a skilled workforce to support autonomous operations.
Long-Term (7-10 Years and Beyond):
- Full realization of the Autonomous Wafer Fab with minimal human intervention.
- Industry-wide standards and best practices for autonomous manufacturing.
- Continuous innovation and refinement of autonomous technologies.
Conclusion
The pathway to the Autonomous Wafer Fab is a transformative journey that holds immense potential for the semiconductor industry. By embracing advanced technologies, fostering collaboration, and investing in the future workforce, fabs can unlock unprecedented levels of efficiency, quality, and flexibility. Autonomous Scheduling Technology, as a key pillar, will play a crucial role in this evolution, driving the industry towards a future where production is seamless, self-optimizing, and truly autonomous. The vision of an Autonomous Wafer Fab is not just a distant possibility but an imminent reality, poised to redefine the landscape of semiconductor manufacturing.
Now available to download: our new Autonomous Scheduling Technology White Paper
We have just released a new White Paper on Autonomous Scheduling Technology (AST) with insights into the latest advancements and benefits.
Click here to read it.