Databricks Runtime 7.5 for ML(不支持)
Databricks於2020年12月發布了這張圖片。
Databricks Runtime 7.5 for Machine Learning為機器學習和數據科學提供了一個現成的環境Databricks Runtime 7.5(不支持).Databricks Runtime ML包含許多流行的機器學習庫,包括TensorFlow、PyTorch和XGBoost。它還支持使用Horovod進行分布式深度學習訓練。
有關更多信息,包括創建Databricks Runtime ML集群的說明,請參見Databricks運行時機器學習.
新特性和主要變化
Databricks Runtime 7.5 ML是建立在Databricks Runtime 7.5之上的。有關Databricks Runtime 7.5(包括Apache Spark MLlib和SparkR)中的新特性的信息,請參見Databricks Runtime 7.5(不支持)發行說明。
Databricks Runtime ML Python環境的主要更改
看到Databricks Runtime 7.5(不支持)參閱Databricks Runtime Python環境的主要更改。有關已安裝的Python包及其版本的完整列表,請參見Python庫.
係統環境
Databricks Runtime 7.5 ML的係統環境與Databricks Runtime 7.5不同:
DBUtils: Databricks Runtime ML不包含庫實用程序(dbutls . Library).你可以使用
%皮普
而且% conda
而不是命令。看到筆記本範圍的Python庫.對於GPU集群,Databricks Runtime ML包含以下NVIDIA GPU庫:
CUDA 10.1更新2
cuDNN 7.6.5
NCCL 2.7.3
TensorRT 6.0.1中
庫
以下部分列出了Databricks Runtime 7.5 ML中包含的不同於Databricks Runtime 7.5的庫。
Python庫
Databricks Runtime 7.5 ML使用Conda進行Python包管理,包括許多流行的ML包。
除了以下章節中Conda環境中指定的包外,Databricks Runtime 7.5 ML還安裝了以下包:
hyperopt 0.2.5.db1
sparkdl 2.1.0-db2
CPU集群上的Python庫
的名字:databricks-ml渠道:-pytorch-違約依賴關係:-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_1 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-補丁= 1.0 = pyhd3eb1b0_2-bcrypt = 3.2.0 = py37h7b6447c_0-布拉斯特區= 1.0 = mkl-信號燈= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.17.1 = h27cfd23_0-ca證書= 2020.10.14 = h06a4308_1#(2021年5月26日維護更新從0更新)-cachetools =以下4.4.1 = py_0-certifi = 2020.11.8 = py37h06a4308_0-cffi = 1.14.0 = py37he30daa8_1#(從2021年5月26日維護更新中的py37h2e261b9_0更新)-chardet = 3.0.4 = py37h06a4308_1003-單擊= 7.0 = py37_0-cloudpickle = 1.4.1 = py_0-configparser = 3.7.4 = py37_0-cpuonly = 1.0 = 0-密碼= 2.8 = py37h1ba5d50_0-周期計= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1裝飾= = py_0-蒔蘿= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未來= 0.18.2 = py37_1-恐嚇= 0.3.3 = py_0-gitdb =你= py_0-gitpython = 3.1.0 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = pyhd3eb1b0_1-isodate = 0.6.0 = py_1-itsdangerous = 1.1.0 = py37_0-絕地= 0.17.2 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.10.0 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.17.1 = h173b8e3_0#(2021年5月26日維護更新從1.16.4更新)-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.3 = he6710b0_2#(2021年5月26日維護更新從3.2.1更新)-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 12.2 = h20c2e04_0#(在2021年5月26日的維護更新中從11.2更新)-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-libuv = 1.40.0 = h7b6447c_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鯖鯊= 1.1.2 = py_0-減價= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h14c3975_1-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_1-忍者= 1.10.2 = py37hff7bd54_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1k = h27cfd23_0#(從2021年5月26日維護更新的1.1.1h更新)-包裝= 20.1 = py_0-熊貓= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.7.0 = py_0-容易受騙的人= 0.5.1 = py37_0-pexpect = 4.8.0 = pyhd3eb1b0_3-pickleshare = 0.7.5 = py37_1001-枕頭= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-情節= 4.12.0 = pyhd3eb1b0_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.6 = py37h3c74f83_1#(2021年5月26日維護更新從2.8.4更新)-ptyprocess = 0.6.0 = pyhd3eb1b0_2-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.8 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = pyhd3eb1b0_1-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_1-python = 3.7.10 = hdb3f193_0#(2021年5月26日維護更新從3.7.6更新)-python-dateutil = 2.8.1發布= py_0-python編輯器的1.0.4 = = py_0-pytorch = 1.7.0 = py3.7_cpu_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 8.1 = h27cfd23_0#(更新自7.0在2021年5月26日維護更新)-= 2.22.0 = py37_1請求-= py_0 1.3.0 requests-oauthlib =版本-重試= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_1-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37h06a4308_0-smmap = 3.0.4 = py_0-sqlite = 3.35.4 = hdfb4753_0#(2021年5月26日維護更新從3.31.1更新)-sqlparse = 0.4.1 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-彙總= 0.8.3 = py37_0-tk = 8.6.10 = hbc83047_0#(2021年5月26日維護更新從8.6.8更新)-torchvision = 0.8.1 = py37_cpu-龍卷風= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-typing_extensions = 3.7.4.3 = py_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-輪= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.5 = h7b6447c_0#(2021年5月26日維護更新從5.2.4更新)-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普:-astunparse = = 1.6.3-azure-core = = 1.9.0-azure-storage-blob = = 12.6.0-databricks-cli = = 0.14.0-diskcache = = 5.1.0-碼頭工人= = 4.4.0-大猩猩= = 0.3.0-horovod = = 0.20.3-joblibspark = = 0.3.0-keras-preprocessing = = 1.1.2-考拉= = 1.4.0-mleap = = 0.16.1-mlflow = = 1.12.1-msr = = 0.6.19-opt-einsum = = 3.3.0-petastorm = = 0.9.7-pyarrow = = 1.0.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-spark-tensorflow-distributor = = 0.1.0-tensorboard = = tripwire-tensorboard-plugin-wit = = 1.7.0-tensorflow-cpu = = 2.3.1-tensorflow-estimator = = tripwire-termcolor = = 1.1.0-xgboost = = 1.2.1 "前綴:/磚/ conda / env / databricks-ml
GPU集群上的Python庫
的名字:databricks-ml-gpu渠道:-pytorch-違約依賴關係:-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_1 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-補丁= 1.0 = pyhd3eb1b0_2-bcrypt = 3.2.0 = py37h7b6447c_0-布拉斯特區= 1.0 = mkl-信號燈= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.17.1 = h27cfd23_0-ca證書= 2020.10.14 = h06a4308_1#(2021年5月26日維護更新從0更新)-cachetools =以下4.4.1 = py_0-certifi = 2020.11.8 = py37h06a4308_0-cffi = 1.14.0 = py37he30daa8_1#(從2021年5月26日維護更新中的py37h2e261b9_0更新)-chardet = 3.0.4 = py37h06a4308_1003-單擊= 7.0 = py37_0-cloudpickle = 1.4.1 = py_0-configparser = 3.7.4 = py37_0-密碼= 2.8 = py37h1ba5d50_0-cudatoolkit = 10.1.243 = h6bb024c_0-周期計= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1裝飾= = py_0-蒔蘿= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未來= 0.18.2 = py37_1-恐嚇= 0.3.3 = py_0-gitdb =你= py_0-gitpython = 3.1.0 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = pyhd3eb1b0_1-isodate = 0.6.0 = py_1-itsdangerous = 1.1.0 = py37_0-絕地= 0.17.2 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.10.0 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.17.1 = h173b8e3_0#(2021年5月26日維護更新從1.16.4更新)-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.3 = he6710b0_2#(2021年5月26日維護更新從3.2.1更新)-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 12.2 = h20c2e04_0#(在2021年5月26日的維護更新中從11.2更新)-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-libuv = 1.40.0 = h7b6447c_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鯖鯊= 1.1.2 = py_0-減價= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h14c3975_1-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_1-忍者= 1.10.2 = py37hff7bd54_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1k = h27cfd23_0#(從2021年5月26日維護更新的1.1.1h更新)-包裝= 20.1 = py_0-熊貓= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.7.0 = py_0-容易受騙的人= 0.5.1 = py37_0-pexpect = 4.8.0 = pyhd3eb1b0_3-pickleshare = 0.7.5 = py37_1001-枕頭= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-情節= 4.12.0 = pyhd3eb1b0_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.6 = py37h3c74f83_1#(2021年5月26日維護更新從2.8.4更新)-ptyprocess = 0.6.0 = pyhd3eb1b0_2-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.8 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = pyhd3eb1b0_1-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_1-python = 3.7.10 = hdb3f193_0#(2021年5月26日維護更新從3.7.6更新)-python-dateutil = 2.8.1發布= py_0-python編輯器的1.0.4 = = py_0-pytorch = 1.7.0 = py3.7_cuda10.1.243_cudnn7.6.3_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 8.1 = h27cfd23_0#(更新自7.0在2021年5月26日維護更新)-= 2.22.0 = py37_1請求-= py_0 1.3.0 requests-oauthlib =版本-重試= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_1-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37h06a4308_0-smmap = 3.0.4 = py_0-sqlite = 3.35.4 = hdfb4753_0#(2021年5月26日維護更新從3.31.1更新)-sqlparse = 0.4.1 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-彙總= 0.8.3 = py37_0-tk = 8.6.10 = hbc83047_0#(2021年5月26日維護更新從8.6.8更新)-torchvision = 0.8.1 = py37_cu101-龍卷風= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-typing_extensions = 3.7.4.3 = py_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-輪= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.5 = h7b6447c_0#(2021年5月26日維護更新從5.2.4更新)-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普:-astunparse = = 1.6.3-azure-core = = 1.9.0-azure-storage-blob = = 12.6.0-databricks-cli = = 0.14.0-diskcache = = 5.1.0-碼頭工人= = 4.4.0-大猩猩= = 0.3.0-horovod = = 0.20.3-joblibspark = = 0.3.0-keras-preprocessing = = 1.1.2-考拉= = 1.4.0-mleap = = 0.16.1-mlflow = = 1.12.1-msr = = 0.6.19-opt-einsum = = 3.3.0-petastorm = = 0.9.7-pyarrow = = 1.0.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-spark-tensorflow-distributor = = 0.1.0-tensorboard = = tripwire-tensorboard-plugin-wit = = 1.7.0-tensorflow = = 2.3.1-tensorflow-estimator = = tripwire-termcolor = = 1.1.0-xgboost = = 1.2.1 "前綴:/磚/ conda / env / databricks-ml-gpu
Java和Scala庫(Scala 2.12集群)
除了Java和Scala庫在Databricks Runtime 7.5, Databricks Runtime 7.5 ML包含以下jar:
CPU集群
組ID |
工件ID |
版本 |
---|---|---|
com.typesafe.akka |
akka-actor_2.12 |
2.5.23 |
ml.combust.mleap |
mleap-databricks-runtime_2.12 |
0.17.3-4882dc3 |
ml.dmlc |
xgboost4j-spark_2.12 |
1.2.0 |
ml.dmlc |
xgboost4j_2.12 |
1.2.0 |
org.mlflow |
mlflow-client |
1.12.1 |
org.scala-lang.modules |
scala-java8-compat_2.12 |
0.8.0 |
org.tensorflow |
spark-tensorflow-connector_2.12 |
1.15.0 |
GPU集群
組ID |
工件ID |
版本 |
---|---|---|
com.typesafe.akka |
akka-actor_2.12 |
2.5.23 |
ml.combust.mleap |
mleap-databricks-runtime_2.12 |
0.17.3-4882dc3 |
ml.dmlc |
xgboost4j-spark-gpu_2.12 |
1.2.0 |
ml.dmlc |
xgboost4j-gpu_2.12 |
1.2.0 |
org.mlflow |
mlflow-client |
1.12.1 |
org.scala-lang.modules |
scala-java8-compat_2.12 |
0.8.0 |
org.tensorflow |
spark-tensorflow-connector_2.12 |
1.15.0 |