The new Native Execution Engine (NEE) leverages columnar format and vectorized processing to boost query execution time.  The performance is driven by translating Spark code into Optimized C++ code with Apache Gluten and Velox.  Utilizing the NEE can increase the speed while reducing cost enabling you to process more.

Scenarios for accelerated performance

• Working with data in parquet and delta formats;
• Handling queries that involve complex transformations and aggregations, benefiting from the engine’s columnar processing and vectorization capabilities;
• Running computationally intensive queries, rather than simple I/O-bound operations.

Balancing Execution vs Optimization

Efficient execution and thorough optimization form two critical pillars of IT project success. While swift execution is essential for meeting immediate deadlines and delivering tangible progress, taking the time to optimize processes and workloads can yield significant long-term cost savings and scalability. Striking the right balance between these priorities ensures that projects remain agile and adaptable, without sacrificing sustainability or quality over time.

1. Execution: Balancing the need for timely execution with adherence to best practices allows projects to meet critical deadlines without compromising long-term goals. By prioritizing execution, teams can achieve tangible progress, while using best practices as adaptable guidelines ensures that quality and sustainability are not sacrificed in the process. This approach fosters agility, enabling teams to refine and optimize workflows iteratively as priorities evolve.
2. Optimize: Prioritizing the optimization of data workloads allows teams to significantly reduce processing costs and durations, enabling more efficient resource allocation and improved operational efficiency. By streamlining workflows and eliminating redundancies, this approach ensures that systems remain scalable and adaptable. Ultimately, optimization fosters a balance between cost-effectiveness and performance, creating a foundation for sustainable and long-term success.