'What is the maximum size for a broadcast object in Spark?

When using Dataframe broadcast function or the SparkContext broadcast functions, what is the maximum object size that can be dispatched to all executors?

apache-sparkdataframeapache-spark-sqlbroadcastsqlsalesforcesql-injectionsoql


Solution 1:[1]

As of Spark 2.4, there's an upper limit of 8 GB. Source Code

Update : The 8GB limit is still valid for Spark 3.2.1 Source Code

Solution 2:[2]

Like mentioned above, the upper limit is 8GB. But when you have several files you want to broadcast, spark push all the data files to driver. The driver join those files & push to to executor nodes. In this process, if the driver's available memory is less than combined broadcast file, you will end up with out of memory error.

Solution 3:[3]

Many SQL injection vulnerabilities are not malicious attacks, and won't result in damage. They just result in unnecessary errors that confuse your users and spoil your application's features.

SELECT * FROM Users WHERE last_name = 'O'Reilly'
                                        ^ mismatched quote

Why would you ever not write your code to avoid SQL injection?

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 Dakshin Rajavel
Solution 3 Bill Karwin

Related Questions