
This was an excellent article by @MarcinRedStone at @redstone_defi. The context of Push vs Pull brings me to my old Operations days where I spent time at Toyota and Amazon.
This post is a reflection of my time and how Push vs Pull is relevant in blockchain discussions, not just Oracle design.
I also share an example from my days at Amazon that is relevant to blockchain design.
Read on.
ORACLE DESIGN AND PUSH VS PULL
Most Oracle design from 2019 is outdated, but the problem is:
1⃣Systems need data when the Oracle is sometimes not ready to give it.
2⃣Systems don't need data at that time, but the Oracle pushes it anyway.
FEAST AND FAMINE PROBLEM
In Queueing Theory, a very common downstream impact from unbalanced systems is the Feast and Famine problem.
In the context of Oracle feeds, we get:
1: Dumb push systems leads to big issues around liquidation.
2: Dumb push systems leads to unnecessary costs that the customer has to bear: more updates, more gas costs.
By using the word "dumb" I'm differentiating from "smart".
A system can have smart push systems, but needs to be responsive to demand, volatility, and characteristics of the queue.
BOTTLENECKS
Which brings me to the main topic I want to discuss, which supports Marcin's thesis--and is borne from my time in manufacturing and graduate school--where I spent time hands-on in very large queueing systems.
I spent years studying and improving systems characterized by a Queue: at Amazon, Toyota, consulting for Disneyland, and Airlines.
Hint: almost all systems have a queue. Some important aspects that are important to remember that highlights the difference between push vs pull:
WHAT IS A BOTTLENECK?
1⃣Every system has a bottleneck.
2⃣A bottleneck is a state of affairs where demand for service exceeds the capacity to serve.
3⃣The Throughput of a system is dependent on the Throughput of the Bottleneck.
4⃣Given (1), (2), & (3), for maximum output, a system ought to keep the bottleneck working at 100% capacity with little or no defects (scrap, waste (muda), time-traps).
5⃣Given (4), Non-bottleneck processes should be working at less than 100% capacity, so as to not over-burden the bottleneck with large batches of work-in-process (WIP).
In Marcin's article, an Oracle that pushes data when the system doesn't need it at the time can be considered work-in-process, or WIP.
Conversely, when the system of DeFi protocol needs the data, but the Oracle is not ready to provide it, it leads to liquidations.
In lean manufacturing nomenclature, WIP is considered one of the 7 Wastes, or Muda, because it doesn't add value to the customer.
The customer in this case is the system that needs the data feed and their customers, the end user who is interacting with the DeFi protocol.
So what's the solution?
DRUM-BUFFER-ROPE
The age-old solution to the feast vs famine problem in systems is the Drum-Buffer-Rope. Visually,
THE DRUM
The Bottleneck or Constraint, acts as a Drum: it sets the rhythm that the whole system should follow. In Lean Manufacturing, this is also called “Takt Time.”
THE BUFFER
There are situations when when upstream processes can’t produce as much as is needed by the Bottleneck.
The result: the Constraint is starved and overall system output is compromised.
So, we must have a buffer of inventory that is the size of the accounted-for variation is demand. This will help to level-out variation. A Buffer will assure that the Constraint never has to wait and, waiting is one of the 7 wastes we want to reduce or eliminate.
Similarly, if upstream processes are producing more than the Constraint has the capacity to handle, then there’s going to be excess inventory sitting in front of the Constraint and, hence, a feast.
💡Put another way, the Buffer is the inventory and inventory is directly related to Lead Time.
(I'll write another post on Little's Law that will help to quantify what I'm talking about that will be interesting to @MaxResnick1 who has a background in operations research and I know thinks about this stuff).
This phenomena is sometimes called the Feast-of-Famine Syndrome.
DBR is used to avoid either of these scenarios, the Feast or the Famine, by dictating the batch size and frequency of the inputs into the Buffer.
THE ROPE
The Rope is a method by which the Constraint can signal to the upstream processes (non-bottleneck processes) when to slow down, when to stop, or when to produce faster and the quantity.
This is called “Pull Scheduling” in Lean Manufacturing terms.
In software data structures, this can be implemented as a data structure called a Stack, with “Push” and “Pop” as the methods for pulling from the Stack.
💡And in Oracle design, it's a flexible Pull method during slow times, and then the Oracle can switch to a Push method during really volatile times. This is where @redstone_defi shines with its modern Oracle design vs its competitors.
AN APPLICATION OF DRUM-BUFFER-ROPE
When I was with Amazon, I led a project where I investigated a production line that was experiencing a Feast/Famine scenario.
There was a lot of waste on this line and it impacted daily production in a serious way. As the team lead, I set out to observe, interview the operators, and collect data on this line. I quantified the cost to Amazon that was a result of the Feast/Famine scenario: costs in terms of actual dollars resulting from missed orders, upgraded orders, overtime of operators, product damage, and safety issues.
Here's what the system looked like:
The distribution you see above is best approximated by the Poisson Distribution, which means that the Mean and the Standard Deviation are approximately the same.
What does this mean?
The picture above graphically shows the Feast scenario: product in totes arrive at the constraint ALL at the same time.
How does this look?
Imagine product and totes everywhere, falling off the conveyor belt, and hundreds of people at the problem area trying to fix it.
Push systems are inherently bad and cause all sorts of issues. Unless the system is a smart push system, but blind or dumb push systems cause all sorts of havoc.
Solution to the above problem?
I led a team of software and industrial engineers that re-engineered this line and we implemented a DBR solution, where the pack-rate at the Constraint would dictate what the upstream pick-rate should be.
In other words, we made sure that the pick-rate would never be higher than the pack-rate. This solution worked and Amazon, was applied to every single operation worldwide, and saved Amazon a lot of money and customers benefit.
Footnote: much of the content above was from an article I wrote about the Theory of Constraints a long time ago for a magazine. I updated it and applied it to Oracle and Blockchain design.