Setting up custom packages on a distributed analytics engine shouldn't slow down your workflow. When you need to install Python libraries on Databricks clusters, choosing the right method depends on whether you are working in a notebook environment, managing libraries across an entire workspace, or deploying custom Python wheels for production jobs.
Databricks runs Python code across a cluster of worker nodes coordinated by a driver node. Because dependencies must be present on every node in the cluster, installing packages requires careful planning rather than simply typing a command into a local terminal.
Workspace Files and Cluster-Scoped Installations
Cluster-scoped libraries are available to every notebook and job attached to a specific cluster. This approach works best for standard data science packages that your team uses consistently across multiple projects.
To install a package at the cluster level, navigate to the compute tab in your Databricks workspace, select your target cluster, and click on the libraries tab. From there, you can install packages directly from PyPI, upload custom Python wheel files, or specify requirements files.
When you install a library via PyPI on a cluster, Databricks automatically distributes the package to all worker nodes during the cluster startup or restart phase. If you add a library to a running cluster, you'll need to restart the cluster for the changes to take effect across all worker nodes. For official documentation on cluster management, consult the Databricks Compute Guide.
Notebook-Scoped Package Management
If you need to test a package quickly or work on an experimental feature without modifying the shared cluster configuration, notebook-scoped libraries offer an effective alternative. Using magic commands or standard Python package installers inside a notebook allows you to install libraries dynamically.
%pip install pandas numpy scikit-learn
Running %pip install inside a notebook cell installs the package on the current Python environment for that specific notebook session. This method isolates your experimental dependencies so they don't interfere with other workloads running on the same cluster.
Keep in mind that notebook-scoped libraries are session-specific. If the cluster restarts or if you detach and reattach the notebook, you'll need to run the installation command again. To manage dependencies reliably across notebook sessions without reinstalling manually every time, you can define your setup in a initialization script or use workspace files. For more details on Python environment management, review the Python Packaging User Guide.
Using Init Scripts for Custom Environments
For complex enterprise deployments that require specific C-dependencies, custom security configurations, or pre-installed packages on every cluster startup, cluster-scoped initialization scripts provide full control.
An init script is a shell script that runs during startup, before any Apache Spark driver or worker JVM starts. You can store these scripts in DBFS (Databricks File System) or workspace files and configure your cluster to execute them automatically.
- Write a bash script that executes your pip commands or environment variables.
- Upload the script to a secure location in DBFS or your workspace.
- Configure the cluster advanced options to point to the init script path.
- Restart the cluster to execute the script across all nodes.
Init scripts ensure that your runtime environment is identical every time the cluster spins up, which prevents unexpected runtime errors in production data pipelines.
Handling Common Library Installation Errors
Installing packages in a distributed computing environment often introduces specific challenges. Recognizing these errors early saves troubleshooting time.
- Version Mismatch: Installing a package that conflicts with the pre-installed Databricks Runtime version can break Spark integrations. Always check the Databricks release notes for your specific runtime version.
- Worker Node Sync Failures: If a library installs on the driver node but fails on worker nodes, jobs running distributed operations will throw module not found errors. Always use cluster-scoped libraries or init scripts to ensure uniform installation across all nodes.
- Network Restrictions: Clusters running inside private VPCs without outbound internet access cannot reach PyPI directly. In these environments, you must configure a local mirror or use pre-bundled wheel files stored in cloud object storage.
To learn more about configuring secure data environments, visit the Apache Software Foundation.
Frequently Asked Questions
Do I need to restart my cluster after installing a PyPI library?
Yes, if you install a library at the cluster level, you must restart the cluster so that the worker nodes download and load the new package into their environment. Notebook-scoped installations via %pip install do not require a cluster restart.
Can I install private Python packages on Databricks?
Yes, you can install private packages by storing wheel files in DBFS, referencing a private PyPI mirror, or configuring your cluster init script to authenticate with a private code repository using secure credentials.
What is the difference between notebook-scoped and cluster-scoped libraries?
Notebook-scoped libraries affect only the active notebook session and are ideal for quick testing. Cluster-scoped libraries are available to all users and notebooks attached to that specific cluster, making them ideal for team environments.
How do I handle conflicting package dependencies?
You can isolate dependencies by using virtual environments where supported, leveraging notebook-scoped installations with explicit version pins, or creating custom cluster runtimes using Docker containers in Databricks.
