The “global router” in the video is not actually one machine. It represents several networking layers working together.
First comes DNS, or the Domain Name System. DNS is like the internet’s phone book. When you type chatgpt.com, DNS translates that name into an IP address that computers can understand.
That IP address can be advertised from many locations using Anycast. With Anycast, the same IP address exists at multiple places around the world, and BGP, or Border Gateway Protocol, helps route your connection toward a suitable nearby network location. So a user in India and a user in Texas may reach different edge locations, even though both opened the same chatgpt.com.
At the edge, servers accept the connection, filter malicious traffic, and protect the service from things like DDoS attacks. Think of the edge as the front door to OpenAI’s infrastructure.
From there, another load-balancing and routing layer decides where the request should actually be processed. That decision can depend on things like geography, available capacity, and the type of GPU or accelerator hardware required.
The request is then sent to the appropriate region, and inside that region, to a suitable compute or GPU cluster.
So when we draw one box called Global Router, that box is really shorthand for:
DNS → Anycast/BGP routing → Edge → Load Balancer → Region → Compute Cluster
The important point is that there is no single global-router machine handling everyone. The routing system itself is distributed across many locations, which prevents that first layer from becoming a bottleneck.
Is this entire system synchronous or asynchronous? I think it’s synchronous- but when I heard the inferences are batched then I am confused. Is response streaming is separated out?
This architecture is not specific to OpenAI, right? This idea of global routing, load balancing, etc exists at many other companies that serve a large number of incoming requests using cloud services and they can also stream their response back at real time.
EXCELLENT SUGGESTION TO FEED AND GOOD EXPLANATION TO MITIGATE ILL EFFECTS OF TECH A LITTLE BIT TO SOME USERS. BUT NOT ENOUGH
Especially use of local information on device or use of personalized content is missing from this.
As this is also relevant on
Other LLM
Google adsense etc Google services
IG FB YouTube
All sorts… Datamining and metadata utilization..
Including cross platform information sharing.. on device information sharing and social emergent stuff people can muster up .
Sharing, utilization, even official business.. healthcare sales..dating life and apps.. reinforcing rich peoples qaly at cost of rudimentary life of poor…etc..
Even if a LLM..add or newsfeed utilizes my private information to my perceived best interest and its only a local loop on my device.. some ill effects need to be considered as a variable..
The “global router” in the video is not actually one machine. It represents several networking layers working together.
First comes DNS, or the Domain Name System. DNS is like the internet’s phone book. When you type chatgpt.com, DNS translates that name into an IP address that computers can understand.
That IP address can be advertised from many locations using Anycast. With Anycast, the same IP address exists at multiple places around the world, and BGP, or Border Gateway Protocol, helps route your connection toward a suitable nearby network location. So a user in India and a user in Texas may reach different edge locations, even though both opened the same chatgpt.com.
At the edge, servers accept the connection, filter malicious traffic, and protect the service from things like DDoS attacks. Think of the edge as the front door to OpenAI’s infrastructure.
From there, another load-balancing and routing layer decides where the request should actually be processed. That decision can depend on things like geography, available capacity, and the type of GPU or accelerator hardware required.
The request is then sent to the appropriate region, and inside that region, to a suitable compute or GPU cluster.
So when we draw one box called Global Router, that box is really shorthand for:
DNS → Anycast/BGP routing → Edge → Load Balancer → Region → Compute Cluster
The important point is that there is no single global-router machine handling everyone. The routing system itself is distributed across many locations, which prevents that first layer from becoming a bottleneck.
No music
no words to describe
Just awesome content! 🔥
So it's like Google but it talks to you 😂
The global router is also need scaling right?
This was very helpful for my CS career.
Underrated content
But still, you miss out on the claude sync with primarily DB, right?
This is really great!
dude, thank you for teaching me this
You can actually use lorax to serve multiple users from a single running model, but at scale I think a different strategy like this is needed
Everyone knows this
Great information!
Is this entire system synchronous or asynchronous? I think it’s synchronous- but when I heard the inferences are batched then I am confused. Is response streaming is separated out?
Pretty crazy that all of this happens in a fraction of a second
But what about their long term memory? How do they do that?
Mcp
Tried phraseroot on text that sounded off It smoothed things enough to keep
Thanks nice info
Neat breakdown!
This architecture is not specific to OpenAI, right? This idea of global routing, load balancing, etc exists at many other companies that serve a large number of incoming requests using cloud services and they can also stream their response back at real time.
Cooool!
Don't know why the algorithm brought me here. I only understood the first 3 seconds. LOL
Nice content though.
Thank you
Bro how did u made this video?
How about likely the question has already been asked before a million times and you can just fetch the answer from cache?
Use it now before prices skyrocket
Pretty cool!
This is amazing🎉
chatgpt have got insanely fast now, good work team🔥.
He missed so many things. Trust me, I used to be IT manager and DevSecOps is my specialty.
😂😂😊
❤
EXCELLENT SUGGESTION TO FEED AND GOOD EXPLANATION TO MITIGATE ILL EFFECTS OF TECH A LITTLE BIT TO SOME USERS. BUT NOT ENOUGH
Especially use of local information on device or use of personalized content is missing from this.
As this is also relevant on
Other LLM
Google adsense etc
Google services
IG
FB
YouTube
All sorts… Datamining and metadata utilization..
Including cross platform information sharing.. on device information sharing and social emergent stuff people can muster up .
Sharing, utilization, even official business.. healthcare sales..dating life and apps.. reinforcing rich peoples qaly at cost of rudimentary life of poor…etc..
Even if a LLM..add or newsfeed utilizes my private information to my perceived best interest and its only a local loop on my device.. some ill effects need to be considered as a variable..
Wait… what exactly do they cache? Isn’t each request’s response unique?
Hey u gota sub buddy
great visuals
Great video but subtitles should be added
But How Do It Know??
I didn't expect that chatgpt only use 1db for writes. I thought it was 3 or more.
Well done
why Sol is unlimited but Opus barely can solve 2 prompts?
What exactly Postgres server holds ? Memory ?
the india servers are routed to the scam email rewrite service to make them sound more american.
But I'm not in India
Stop.
ChattyG
👍
Distributed Systems was fun to learn in Uni