// INTRODUCTION & ABSTRACT

Vectra: The Quarantine Matrix

Constraining Neural Hallucinations in 3D Gaussian Environments

The hyper-accelerated rise of generative artificial intelligence is rewiring the rules of 3D content creation. Yet, seamlessly jacking everyday 2D inputs—like text prompts and flat images—into fully interactive, dynamic 3D constructs remains a critical bottleneck. This paper introduces an end-to-end framework engineered to generate, extract, and simulate high-fidelity 3D objects directly from simple visual and textual data. By harnessing advanced neural rendering and spatial splatting algorithms, our system spins up robust 3D assets on the fly, entirely bypassing the tedious grind of traditional manual modeling. We orchestrate a streamlined pipeline that fuses zero-shot semantic extraction, generative mesh synthesis, and web-based physics integration. This unified architecture doesn't just supercharge the rendering of complex 3D scenes; it breathes real-time kinetic life into them, enabling fluid dynamic simulation and direct user manipulation. We benchmark the framework’s performance across structural integrity, pipeline latency, and interactive immersion within a scalable network. Ultimately, this work delivers a highly optimized, plug-and-play solution that accelerates the 3D creation workflow, paving the way for the next generation of accessible, dynamic, and fully interactive digital realities.

AUTHOR: PARSA BESHARAT

|

TECHNISCHE UNIVERSITÄT BERGAKADEMIE FREIBERG, GERMANY

// SECTION_01 // RELATED WORK FOUNDATIONS

Mathematical Foundations

Mildenhall et al. (2020) proposed modeling physical environments as continuous 5D functions using a neural network MLP $F_\Theta : (x, d) \to (c, \sigma)$ to calculate density $\sigma$ and emitted color $c$. Volume accumulation is driven by differentiable numerical ray integration. Quality standards utilize Peak Signal to Noise Ratio (PSNR) to audit variance, Structural Similarity Index Measure (SSIM) to check contrast, luminance, and structural covariance, and Learned Perceptual Image Patch Similarity (LPIPS) to evaluate deep perceptual features.

// Volume Rendering Integration & Stratified Sampling:

C(\mathbf{r}) = \int_{t_n}^{t_f} T(t) \sigma(\mathbf{r}(t)) \mathbf{c}(\mathbf{r}(t), \mathbf{d}) dt, \quad t_i \sim \mathcal{U} \left[ t_n + \frac{i-1}{N}(t_f - t_n), \; t_n + \frac{i}{N}(t_f - t_n) \right]

// Peak Signal to Noise Ratio:

PSNR(I) = 10 \cdot \log_{10} \left( \frac{MAX(I)^2}{MSE(I)} \right)

// Structural Similarity:

SSIM(x, y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)}

// Learned Perceptual Similarity (LPIPS):

LPIPS(x, y) = \sum_{l=1}^L \frac{1}{H_l W_l} \sum_{h,w} ||w_l (x_{lhw} - y_{lhw})||_2^2

// VISUAL ARCHIVE // FIGURE 1

// Figure 1: NeRF Scene Representation & Differentiable Volume Rendering (Mildenhall et al., 2020)

// VISUAL ARCHIVE // FIGURE 6

// Figure 6: Comparisons on Image-to-3D. Evaluating the balance between generation speed and mesh quality.

// SECTION_03 // SEMANTIC CONSISTENCY

DietNeRF and Semantic Regularization

Standard NeRF daemons suffer catastrophic geometry collapse when optimized under few-shot viewpoints. Without global constraints, $\mathcal{L}_{MSE}$ minimization leads to degenerate near-field "floaters" packed against virtual cameras. DietNeRF (Jain et al., 2021) bypasses this vulnerability by enforcing a semantic consistency loss. It compares high-level invariant representations of synthetic renders against pre-observed poses using normalised Vision Transformer (ViT) embeddings.

// Semantic Consistency L2 Loss:

\mathcal{L}_{SC, \ell_2}(I, \hat{I}) = \frac{\lambda}{2} \| \phi(I) - \phi(\hat{I}) \|_2^2

// Cosine Similarity formulation:

\mathcal{L}_{SC}(I, \hat{I}) = \lambda \phi(I)^T \phi(\hat{I})

// VISUAL ARCHIVE // FIGURE 3

// Figure 3: DietNeRF Novel Views synthesized from sparse input DTU dataset (Jain et al., 2021)

// VISUAL ARCHIVE // FIGURE 2

// Figure 2: View Synthesis comparison on Realistic Synthetic Dataset (Jain et al., 2021)

// SECTION_05 // GENERATIVE 3D

Generative & Dynamic Gaussian Splatting

DreamGaussian (Tang et al., 2024): Transitions generative 3D from neural fields to explicit 3D Gaussian nodes tracking coordinate parameters $\Theta_i = \{x_i, s_i, q_i, \alpha_i, c_i\}$. High-speed optimization utilizes Score Distillation Sampling (SDS) with 2D diffusion noise gradients, followed by mesh extraction from volumetric density $d(\mathbf{x})$ and UV-space texture refinement.

Dynamic3D (Luiten et al., 2023): Reconstructs moving environments chronologically, optimizing kinematics and temporal rotations $R_{i,t}$. Enforces rigidity ($\mathcal{L}_{\text{rigid}}$), rotational ($\mathcal{L}_{\text{rot}}$), and isometry ($\mathcal{L}_{\text{iso}}$) loss functions to prevent spatial tracking drift.

LGM (Tang et al., 2024): Synthesizes high-resolution 3D Gaussians in a single forward pass. It anchors multi-view orbital images using 9-channel Plücker ray embeddings $f_i = \{c_i, o_i \times d_i, d_i\}$ fed into an Asymmetric U-Net with cross-view self-attention.

TRELLIS.2 (Xiang et al., 2025): Employs structured voxels (O-Voxel) to represent geometry and material tuples $f =\{(f^{shape}_i, f^{mat}_i, p_i)\}_{i=1}^L$ on a flexible dual grid via Quadratic Error Function (QEF) minimization, compressed using a sparse-convolutional autoencoder (SC-VAE).

// Score Distillation Sampling (SDS) Loss:

\nabla_\Theta \mathcal{L}_{SDS} = \mathbb{E}_{t,p,\epsilon} \left[ w(t) \left( \epsilon_\phi(I^p_{RGB}; t, \tilde{I}^r_{RGB}, \Delta p) - \epsilon \right) \frac{\partial I^p_{RGB}}{\partial \Theta} \right]

// TRELLIS.2 Dual Grid QEF Minimization:

\min_{v\in voxel} e(v) = \sum_i d^2_{\Pi,i} + \lambda_{bound} \sum_j d^2_{L,j} + \lambda_{reg} d^2_{\hat{q}}

// VISUAL ARCHIVE // FIGURE 7

// Figure 7: Architecture of LGM asymmetric U-Net with cross-view self-attentions (Tang et al., 2024)

// VISUAL ARCHIVE // FIGURE 13

// Figure 13: Sequential VRAM Lifecycle on restricted GPU VRAM ceiling

// JavaScript: executePromptSummon()

function executePromptSummon() {
    const prompt = creatorPromptInput.value.trim();
    if (!prompt) return;
    showCreatorLoader('Connecting to Stable Diffusion SD3 generator...', 15);
    
    fetch('/api-proxy/summon', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ prompt: prompt })
    })
    .then(response => {
        if (!response.ok) throw new Error(`HTTP ${response.status}`);
        showCreatorLoader('Image generated. Forging 3D geometry locally via TripoSR...', 55);
        return response.arrayBuffer();
    })
    .then(arrayBuffer => {
        showCreatorLoader('Reconstruction complete. Loading 3D spatial mesh...', 85);
        const loader = new GLTFLoader();
        loader.parse(arrayBuffer, '', (gltf) => {
            const model = gltf.scene;
            const forward = new THREE.Vector3();
            camera.getWorldDirection(forward);
            const targetPos = new THREE.Vector3().copy(camera.position).addScaledVector(forward, 5);
            model.position.copy(targetPos);
            if (viewer && viewer.splatMesh) viewer.splatMesh.visible = false;
            scene.add(model);
            hideCreatorLoader();
        });
    })
    .catch(err => {
        console.error('SYS_ERR: Summon failed:', err.message);
        hideCreatorLoader();
    });
}

// Scenario 2 Asynchronous Client-Side Orchestration

// SECTION_07 // METHODOLOGY BLUEPRINTS

Decoupled Spatial Architecture

The Vectra Spatial Computing Protocol is engineered on a strictly decoupled client-server architecture. The browser-based Client Presentation Layer renders the 3D Gaussian environment via gsplat.js and calculates collision physics using Cannon.js. The lightweight frontend captures user selections or prompts without executing any neural network operations, ensuring non-blocking real-time rendering.

The autonomous Local GPU Forge (Backend) runs a FastAPI server hosting the neural architectures: U2Net for semantic masking, SDXL-Lightning for rapid text-to-image synthesis, and TripoSR for volumetric reconstruction.

// Figure 10: Asynchronous high-level topology of the Vectra Protocol

// Client-Side Zwicker Splat Projection:

\Sigma_{2D} = J E \Sigma E^T J^T, \quad \mu_{2D} = K \left( \frac{E\mu}{(E\mu)_z} \right)

Deep Splat Excavation (DBSE) handles Z-fighting and intersection clipping by raycasting a 3D bounding box from the client viewport and overriding intersecting Gaussian splat opacities to zero.

// METHODOLOGY VISUALIZATION // FLOWCHART

Vectra Spatial Processing Map

Flow Telemetry: Processing Active Segment...

// VISUAL ARCHIVE // FIGURE 11

// Figure 11: Generative Extraction Pipeline (Scenario 1) Flowchart

// JavaScript: captureSelectionSnapshot()

function captureSelectionSnapshot(bb) {
    try {
        const prevBg = scene.background ? scene.background.clone() : null;
        const prevFog = scene.fog;
        const prevGridVisible = gridHelper.visible;
        
        scene.background = new THREE.Color(0xffffff);
        scene.fog = null;
        gridHelper.visible = false;
        renderer.render(scene, camera);

        const srcCanvas = renderer.domElement;
        const rect = srcCanvas.getBoundingClientRect();
        const scaleX = srcCanvas.width / rect.width;
        const scaleY = srcCanvas.height / rect.height;
        const clampedW = Math.min(Math.round(bb.w * scaleX), srcCanvas.width);
        const clampedH = Math.min(Math.round(bb.h * scaleY), srcCanvas.height);

        const offCanvas = document.createElement('canvas');
        offCanvas.width = clampedW;
        offCanvas.height = clampedH;
        const offCtx = offCanvas.getContext('2d');
        offCtx.drawImage(srcCanvas, Math.round((bb.x - rect.left) * scaleX), Math.round((bb.y - rect.top) * scaleY), clampedW, clampedH, 0, 0, clampedW, clampedH);
        const dataURL = offCanvas.toDataURL('image/png');

        scene.background = prevBg;
        scene.fog = prevFog;
        gridHelper.visible = prevGridVisible;
        renderer.render(scene, camera);

        fetch('/api-fastapi/extract', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ image: dataURL })
        }).then(response => response.arrayBuffer())
          .then(arrayBuffer => {
              const loader = new GLTFLoader();
              loader.parse(arrayBuffer, '', (gltf) => {
                  scene.add(gltf.scene);
              });
          });
    } catch (err) {
        console.error('[SYSTEM_ERR] Snapshot failed:', err.message);
    }
}

// Scenario 1 Client-Side Viewport Slicing

// VISUAL ARCHIVE // FIGURE 12

// Figure 12: Deep Splat Excavation (DBSE) Frustum Raycasting & Masking

// Algorithm 1: DBSE Viewport Slicing & Injection

Require: User selection box $B_{2D} = (x, y, w, h)$, Camera Matrix $C$, 3D Scene $S$

Ensure: Transmitted viewport slice $I_{crop}$ and injected 3D mesh $M_{glb}$

1: Initialize interactive 2D canvas overlay linked to mouse coordinates

2: If Mouse drag event detected then

3: Update $B_{2D} \leftarrow$ dynamic $(x, y, w, h)$ bounds

4: Render selection rectangle on overlay

5: End If

6: If Mouse release event detected then

7: Disable $S$ environmental variables (fog, grid, ambient light)

8: Render clean, unobstructed frame of $S$

9: Map $B_{2D}$ bounds to absolute WebGL coordinates $(p_x, p_y, p_w, p_h)$

10: $I_{crop} \leftarrow$ Extract pixels from canvas buffer at mapped bounds

11: Restore $S$ environmental variables

12: Transmit $I_{crop}$ payload to /api-fastapi/extract via POST

13: Await HTTP Response $\rightarrow$ ArrayBuffer $A_{glb}$

14: $M_{glb} \leftarrow$ Parse GLTF geometry from $A_{glb}$

15: Calculate target centroid $C_{target}$ derived from $C$ forward vector

16: Translate $M_{glb}$ to $C_{target}$ and apply dimension normalization

17: Inject $M_{glb}$ into $S$ and update rendering loop

18: End If

// Localized Depth & Masking Logic

// CONCLUSION SUMMARY

Conclusion

As spatial computing and generative artificial intelligence converge, the necessity for robust, secure, and highly optimized integration architectures becomes paramount. The Vectra Spatial Computing Protocol successfully bridges the gap between high-fidelity digital twins and localized generative AI pipelines. By enforcing a decoupled, asynchronous client-server architecture, it isolates the visual presentation layer from the heavy tensor operations of the inference backend, ensuring that real-time spatial navigability (maintaining 30–60 FPS) is never compromised by computational latency on constrained edge hardware (strictly within an 8GB VRAM threshold).

Furthermore, the Deep Splat Excavation (DBSE) algorithm resolves the critical spatial occlusion problem via non-destructive, shader-level volumetric masking, allowing synthetic assets to be surgically injected into digitized spaces without permanent alteration of the point cloud. This lays the groundwork for embedding definitive mathematical safeguards, such as Control Barrier Functions (CBFs), directly in the spatial rendering pipeline to guarantee physical alignment and collision security.

// CITATIONS & PORTAL

References & Launch

[1] Mildenhall et al. (2020) - NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV).
[2] Rabby & Zhang (2024) - BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fields (arXiv).
[3] Jain et al. (2021) - Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis (ICCV).
[4] Niemeyer et al. (2022) - RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (CVPR).
[5] Tang et al. (2024) - DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (ICLR).
[6] Luiten et al. (2023) - Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (arXiv).
[7] Tang et al. (2024) - LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (arXiv).
[8] Xiang et al. (2025) - Native and Compact Structured Latents for 3D Generation (arXiv).
[9] He et al. (2025) - SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modeling (arXiv).

[ LAUNCH PORTAL CORE ]