"explorable" and "immersive" is definitely a bold choice of words when you can't really get below the level of the buildings before the gaussian splatting is very obvious. Sure, it's impressive that you can get that detailed from a few satellite images, but I think that might be overselling it a bit
We probably won't. GS is a reconstructive method, so when data is unavailable, you can only perform poor interpolation. You would need additional generative, not reconstructive, models. However, this would open the door to unfaithful augmentation again.
You’d be surprised what GIS - or at least GIS - adjacent customers want. If you think about any cute-but-useless map detail that comes to your mind there is likely a paying customer for it.
If you want to dig more, I suspect Microsoft got a lot (if not all) of it from blackshark.ai . They're one of the companies whose logo shows up during game start.
I knew their name because, when I worked for an Airbus subsidiary, we talked with them about a solution to generate 3D environments for any/every airport.
They had some cool stuff but also some wonky stuff at the time (like highway overpasses actually being rendered as walls across the highway).
This is so cool. I used to work on urban heat island analysis and now work in natural catastrophe modelling, and in both cases knowing the average heights/volumes of buildings is a very handy thing to have but is surprisingly difficult information to retrieve. Even a coarse estimate available at annual resolution has some really awesome use cases, very excited to see this.
Maybe dumb question but how do I just take a sat image and create the scene? The scripts in the repo are all about training which I assume requires you to have the 3d data too.
These sort of projects always look cool but I think the real "wow factor" would be a file upload where you can see the result on your image. I assume there are reasons why this isn't done.
Do you mean converting your image into a 3D scene?
This is where we were heading with our 3D volumetric video company https://ayvri.com
We were working on blending 3D satellite imagery with your ground view (or low flying in the case of paragliders) photos and videos to create a 3D scene.
Our technology was acquired prior to us being able to fully realize the vision (and we moved on to another project).
MSFS 2024 already does photogrammetry from satellite photos. However, it builds triangle geometry much like is done from aerial photography, because gaussian splats are not suitable for games; you can't build collision geometry from a gaussian splat for example.
Nice, but when you look up close things like this and Google Earth look like a post-apocalyptic scene :)
It would be amazing if they could also take user-generated photos and videos at ground level and accurate mapping data (that has building outlines) and clean that up to something presentable.
I mean, what they do here is what google and apple are already doing for years. It's time for the next step.
I suspect hybrid solutions will remove the limitations of GS, with (eventually...) some smooth hand off. Do clean-enough GS like this; then hand the output to other systems which covert into forms more useful for your application and which adopt e.g. textures from localized photos etc.
In fact you wouldn't even need to be limited to earth. Why not throw in Google Moon and steal a moon buggy while shooting scientific rovers and doing cool flips out of craters?
There are already 3D globes of the moon, for example with cesiumjs: https://sandcastle.cesium.com/?id=moon you can even import it to unity and other engines
"explorable" and "immersive" is definitely a bold choice of words when you can't really get below the level of the buildings before the gaussian splatting is very obvious. Sure, it's impressive that you can get that detailed from a few satellite images, but I think that might be overselling it a bit
We're early days. Models will soon interpolate all of that. Eventually in real time.
I wouldn't knock the research. The results look impressive to me.
We probably won't. GS is a reconstructive method, so when data is unavailable, you can only perform poor interpolation. You would need additional generative, not reconstructive, models. However, this would open the door to unfaithful augmentation again.
Different applications.
GIS won't want generative hallucinations.
Consumer mapping apps, social applications, and games (eg. flight sims) will want the maps to look as good as possible.
GIS don't want half exploded buildings either. Nor would they care about photographic textures on the 3D models.
You’d be surprised what GIS - or at least GIS - adjacent customers want. If you think about any cute-but-useless map detail that comes to your mind there is likely a paying customer for it.
While the methodology wasn't published, Microsoft did something similar for Flight Simulator.
https://www.smithsonianmag.com/air-space-magazine/flight-box...
If you want to dig more, I suspect Microsoft got a lot (if not all) of it from blackshark.ai . They're one of the companies whose logo shows up during game start.
I knew their name because, when I worked for an Airbus subsidiary, we talked with them about a solution to generate 3D environments for any/every airport.
They had some cool stuff but also some wonky stuff at the time (like highway overpasses actually being rendered as walls across the highway).
This is so cool. I used to work on urban heat island analysis and now work in natural catastrophe modelling, and in both cases knowing the average heights/volumes of buildings is a very handy thing to have but is surprisingly difficult information to retrieve. Even a coarse estimate available at annual resolution has some really awesome use cases, very excited to see this.
Maybe dumb question but how do I just take a sat image and create the scene? The scripts in the repo are all about training which I assume requires you to have the 3d data too.
These sort of projects always look cool but I think the real "wow factor" would be a file upload where you can see the result on your image. I assume there are reasons why this isn't done.
Do you mean converting your image into a 3D scene?
This is where we were heading with our 3D volumetric video company https://ayvri.com
We were working on blending 3D satellite imagery with your ground view (or low flying in the case of paragliders) photos and videos to create a 3D scene.
Our technology was acquired prior to us being able to fully realize the vision (and we moved on to another project).
Is there any reason that this couldn't integrate Street View data?
This would be the next step for flight simulators, which while remarkable still require handmade assets for accurate details.
MSFS 2024 already does photogrammetry from satellite photos. However, it builds triangle geometry much like is done from aerial photography, because gaussian splats are not suitable for games; you can't build collision geometry from a gaussian splat for example.
afaik msfs uses partially automatically generated 3d assets (from Bing Maps?) from aerial/satellite imagery.
Very cool; interesting how it turns all the trees into puffballs though. Some artifact of the pre-trained depth estimation or diffusion model maybe?
This could be specially good for a world 3d model for flightgear.
Once Flightgear could use Google Earth assets.
Nice, but when you look up close things like this and Google Earth look like a post-apocalyptic scene :)
It would be amazing if they could also take user-generated photos and videos at ground level and accurate mapping data (that has building outlines) and clean that up to something presentable.
I mean, what they do here is what google and apple are already doing for years. It's time for the next step.
> I mean, what they do here is what google and apple are already doing for years
This is gaussian splatting. I'm pretty confident that google/apple have not done that.
Oh I didn't realise it was a different technique but the result is similarly bad when zoomed in :(
Re: utility in games,
I suspect hybrid solutions will remove the limitations of GS, with (eventually...) some smooth hand off. Do clean-enough GS like this; then hand the output to other systems which covert into forms more useful for your application and which adopt e.g. textures from localized photos etc.
It's just a bit of engineering and compute...
Now the GTA: Anywhere please...
Ya, can't wait to play GTA: Nova Zemyla.
In fact you wouldn't even need to be limited to earth. Why not throw in Google Moon and steal a moon buggy while shooting scientific rovers and doing cool flips out of craters?
There are already 3D globes of the moon, for example with cesiumjs: https://sandcastle.cesium.com/?id=moon you can even import it to unity and other engines
[flagged]