TensorFlow. Python

De wikijoan
Salta a la navegació Salta a la cerca

Instal·lació (gener 2020)

TensorFlow 2 packages are available

  • tensorflow —Latest stable release with CPU and GPU support (Ubuntu and Windows)
  • ptf-nightly —Preview build (unstable). Ubuntu and Windows include GPU support.

1. Install the Python development environment on your system 

$ python3 --version
Python 3.6.9

$ pip --version (però compte! aquest és el pip2)
pip 19.3.1 from /home/joan/.local/lib/python2.7/site-packages/pip (python 2.7)

$ pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
$ pip3 install --upgrade pip
Collecting pip
...
Installing collected packages: pip
Successfully installed pip-20.0.2

$ sudo pip3 install -U virtualenv

$ virtualenv --version
16.7.9

2. Create a virtual environment (recommended)

virtualenv is a tool to create isolated Python environments

Create a new virtual environment by choosing a Python interpreter and making a ./venv directory to hold it: 

$ virtualenv --system-site-packages -p python3 ./venv
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/joan/venv/bin/python3
Also creating executable in /home/joan/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

Activate the virtual environment using a shell-specific command: 

$ source ./venv/bin/activate

When virtualenv is active, your shell prompt is prefixed with (venv). El prompt queda: 

(venv) joan@joanHP:~$


Install packages within a virtual environment without affecting the host system setup. Start by upgrading pip: 

(venv) $ pip install --upgrade pip

(venv) $ pip list  # show packages installed within the virtual environment
Package               Version  
--------------------- ---------
apt-clone             0.2.1    
apturl                0.5.2    
beautifulsoup4        4.6.0    
Brlapi                0.6.6
...

And to exit virtualenv later: 

(venv) $ deactivate  # don't exit until you're done using TensorFlow

3. Install the TensorFlow pip package Choose one of the following TensorFlow packages to install from PyPI:

  • tensorflow —Latest stable release with CPU and GPU support (Ubuntu and Windows).
  • tf-nightly —Preview build (unstable). Ubuntu and Windows include GPU support.
  • tensorflow==1.15 —The final version of TensorFlow 1.x.

Instal·larem el primer paquet. 

(venv) $ pip install --upgrade tensorflow

Verify the install: 

$ python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

En comptes d'instal·lar-ho en l'entorn virtual també ho puc instal·lar en el sistema: 

$ pip3 install --user --upgrade tensorflow  # install in $HOME

Però el problema és el mateix: 

$ python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
2020-01-30 17:08:16.519868: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory

El TensorFlow el tinc ben instal·lat: 

$ python3 -m pip freeze | grep tensor
tensorboard==2.1.0
tensorflow==2.1.0
tensorflow-estimator==2.1.0

però el problema que tinc és el mateix que es documenta aqui:

Des de l'entorn virtual: 

(venv)$ pip install tensorflow-cpu
...
Successfully installed tensorflow-cpu-2.1.0

I ara sí que funciona: 

(venv) $ python -c "import tensorflow as tf;"

(venv) $ python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

2020-01-30 17:22:09.073469: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2394605000 Hz
2020-01-30 17:22:09.074097: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x48ae480 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-30 17:22:09.074166: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
tf.Tensor(369.16412, shape=(), dtype=float32)

I ara ho instal·lem directament al sistema, sense l'entorn virtual (no caldria): 

$ pip3 install tensorflow-cpu
...
Successfully installed tensorflow-cpu-2.1.0

$ python3 -c "import tensorflow as tf;"

$ python3 -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

TensorFlow Tutorial For Beginners (Belgium traffic signals) (prova 1)

NOTA: el problema és que aquest tutorial està fent amb TF1 (tot i que és bastant nou), però ara ja es va pel TF2, i de fet quan he fet la instal·lació m'ha quedat instal·lat el TF2, i m'està donant molts problemes. Conclusió: més val centrar-se i mirar un tutorial i exemples de TF2.

Les dades:

Al Github:

El projecte està a ~/projectes/tensorflow_tutorial, i creem un entorn virtual en aquest directori: 

$ cd ~/projectes/tensorflow_tutorial
$ virtualenv --system-site-packages -p python3 ./venv
Already using interpreter /usr/bin/python3
...
done.
$ source ./venv/bin/activate
(venv) $ PS1="E$ "
E$

E$ python --version
Python 3.6.9

Dependències per al projecte: 

E$ python belgium_traffic_signs_v1.py 
NameError: name 'skimage' is not defined

$ pip3 install scikit-image

E$ python belgium_traffic_signs_v1.py 
NameError: name 'skimage' is not defined

Problema a la línia 28: 

images.append(skimage.data.imread(f))

El problema no és que la llibreria no estigui ben instal·lada

El que tinc instal·lat: 

$ python3 -m pip freeze | grep tensor
tensorboard==2.1.0
tensorflow==2.1.0
tensorflow-cpu==2.1.0
tensorflow-estimator==2.1.0
tensorflow-gpu==2.1.0

TensorFlow Tutorial For Beginners (Belgium traffic signals) (prova 2)

python i python3, en aquest entorn virtual, és el mateix: 

E $ python --version
Python 3.6.9
E $ python3 --version
Python 3.6.9

E $ pip install tensorflow
...
Successfully installed tensorboard-2.1.0 tensorflow-2.1.0 tensorflow-estimator-2.1.0

Per veure què se'ns ha instal·lat i quines versions: 

E $ python3 -m pip freeze | grep tensor
tensorboard==2.1.0
tensorflow==2.1.0
tensorflow-estimator==2.1.0

E $ python -c "import tensorflow as tf;"
2020-02-03 13:33:29.725233: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-03 13:33:29.725330: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-03 13:33:29.725346: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly

Hi ha dos problemes. Un és relacionat amb libnvinfer.so.6, i l'altre és un problema amb les targetes gràfiques NVIDIA.

-> hi ha algun problema amb els drivers de la targeta gràfica, em sembla.

Però per fer aquest tutorial necessitor tensorflow-1.15, que no és l'últim (estan havent-hi bastants canvis de tensorflow 1 a 2).

Però si vull instal·lar tensorflow-1.15: 

E $ pip install tensorflow==1.15
Collecting tensorflow==1.15
...
    Uninstalling tensorflow-2.1.0:
      Successfully uninstalled tensorflow-2.1.0
...

E $ python3 -m pip freeze | grep tensor

tensorboard==1.15.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1

E $ python3 -c "import tensorflow as tf;"

i ara no em dóna el missatge emprenyador 

E $ python prova2.py 
WARNING:tensorflow:From prova2.py:12: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

funciona, tot i que protesta. Estic treballant amb 1.15. Però en la versió 2 no existeixen les Sessions (protesta tot i que treballo amb la versió 1) Puc treballar amb Sessions.

GPU support i CUDA enabled cards

Software requirements

The following NVIDIA® software must be installed on your system:

  • NVIDIA® GPU drivers —CUDA 10.1 requires 418.x or higher.
  • CUDA® Toolkit —TensorFlow supports CUDA 10.1 (TensorFlow >= 2.1.0)
  • CUPTI ships with the CUDA Toolkit.
  • cuDNN SDK (>= 7.6)
  • (Optional) TensorRT 6.0 to improve latency and throughput for inference on some models.

Però el portàtil no té una targeta NVIDIA, sinó senzillament una targeta gràfica intel: 

$ lspci -v
...
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) (prog-if 00 [VGA controller])
	Subsystem: Hewlett-Packard Company 2nd Generation Core Processor Family Integrated Graphics Controller
	Flags: bus master, fast devsel, latency 0, IRQ 27

Què és CUDA? En poques paraules, CUDA seria la possibiltiat d'utilitzar la CPU de la gràfica per fer operacions matemàtiques. I és per aixó que CUDA és important.

Aclaració, important

Si no tinc una targeta gràfica NVIDIA, és inútil instal·lar tensorflow-gpu i CUDA. 

$ lspci | grep Graph
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)

$ lspci | grep graph

Per tant, oblidar-se d'aquest tema.

En el cas de què es disposi una gràfica NVIDIA, la idea és que es fa servir per tal de què els càlculs matemàtics vagin més ràpid, doncs el ML necessita processament de càlcul.

Tutorials oficials

E $ pip install tensorflow
E $ python3 -m pip freeze | grep tensor
tensorboard==2.1.0
tensorflow==2.1.0
tensorflow-estimator==2.1.0

from __future__ import absolute_import, division, print_function, unicode_literals

# Install TensorFlow

import tensorflow as tf

E $ python prova1.py 
2020-02-03 14:29:09.335379: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-03 14:29:09.335477: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-03 14:29:09.335493: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly

Instal·lar tensorflow amb contenedors Docker

Docker uses containers to create virtual environments that isolate a TensorFlow installation from the rest of the system. TensorFlow programs are run within this virtual environment that can share resources with its host machine (access directories, use the GPU, connect to the Internet, etc.). The TensorFlow Docker images are tested for each release.

Docker is the easiest way to enable TensorFlow GPU support on Linux since only the NVIDIA® GPU driver is required on the host machine (the NVIDIA® CUDA® Toolkit does not need to be installed).

Aquests són els problemes que he tingut. A veure si amb Docker aconsegueixo tenir un entorn estable per fer proves amb tensorflow.

TensorFlow Docker requirements:

  1. Install Docker on your local host machine: https://docs.docker.com/install/
  2. For GPU support on Linux, install NVIDIA Docker support. -> aquesta part NO l'he de fer, jo no tinc una targeta NVIDIA.

Take note of your Docker version with docker -v. Versions earlier than 19.03 require nvidia-docker2 and the --runtime=nvidia flag. On versions including and after 19.03, you will use the nvidia-container-toolkit package and the --gpus all flag. Both options are documented on the page linked above. 

$ cat /etc/os-release 
NAME="Linux Mint"
VERSION="19.2 (Tina)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 19.2"
VERSION_ID="19.2"
HOME_URL="https://www.linuxmint.com/"
SUPPORT_URL="https://forums.ubuntu.com/"
BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
PRIVACY_POLICY_URL="https://www.linuxmint.com/"
VERSION_CODENAME=tina
UBUNTU_CODENAME=bionic

Tinc Linux Mint (Tina). La versió equivalent de Ubuntu és bionic. 

$ sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable'

$ sudo apt-get install docker-ce docker-ce-cli containerd.io

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
Digest: sha256:9572f7cdcee8591948c2963463447a53466950b3fc15a247fcad1917ca215a2f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

For GPU support on Linux, install NVIDIA Docker support: -> aquesta part jo NO l'he de fer. 

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
joan@joanHP:~/projectes/tensorflow_tutorial$ echo $distribution 
linuxmint19.2

$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -

$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
# Unsupported distribution!
# Check https://nvidia.github.io/nvidia-docker

$ sudo apt-get update

Solució per a Linux Mint: 

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update

$ sudo apt-get install -y nvidia-container-toolkit
S'està llegint la llista de paquets… Fet 
S'està construint l'arbre de dependències       
S'està llegint la informació de l'estat… Fet
E: No s'ha trobat el paquet nvidia-container-toolkit

No puc instal·lar el NVIDIA Docker support perquè estic utilitzant Linux Mint, i no li agrada. (Però bàsicament és perquè jo no tinc una targeta NVIDIA). 

$ sudo systemctl restart docker

Download a TensorFlow Docker image

Un cop he instal·lat Docker, ara m'he de descarregar la imatge de tensorflow.

Repositori de les imatges: 

*latest	The latest release of TensorFlow CPU binary image. Default.
*nightly	Nightly builds of the TensorFlow image. (unstable)
*version	Specify the version of the TensorFlow binary image, for example: 2.1.0
*devel	Nightly builds of a TensorFlow master development environment. Includes TensorFlow source code.

Each base tag has variants that add or change functionality: 

Tag Variants	Description
*tag-gpu	The specified tag release with GPU support. (See below)
*tag-py3	The specified tag release with Python 3 support.
*tag-jupyter	The specified tag release with Jupyter (includes TensorFlow tutorial notebooks)

Descarreguem la latest stable release: 

$ sudo docker pull tensorflow/tensorflow 
[sudo] contrasenya per a joan:         
Torneu-ho a provar
[sudo] contrasenya per a joan:         
Using default tag: latest
latest: Pulling from tensorflow/tensorflow
2746a4a261c9: Pull complete 
4c1d20cdee96: Pull complete 
0d3160e1d0de: Pull complete 
c8e37668deea: Pull complete 
e52cad4ccd83: Pull complete 
fac2cc420193: Pull complete 
3cae0d3239d1: Pull complete 
d383bce69b6c: Pull complete 
8a90b3d85a18: Pull complete 
4baf29878469: Pull complete 
4475017ba949: Pull complete 
Digest: sha256:cec133a881c752a82953e88f34a1e9f878b5523779666528c5d2bdfefe3ba6ae
Status: Downloaded newer image for tensorflow/tensorflow:latest
docker.io/tensorflow/tensorflow:latest

$ sudo docker run -it --rm tensorflow/tensorflow    python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
2020-02-03 15:41:29.474266: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-03 15:41:29.474925: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-03 15:41:29.475005: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-02-03 15:41:32.640011: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-03 15:41:32.640055: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-03 15:41:32.640090: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (89bb3e229959): /proc/driver/nvidia/version does not exist
2020-02-03 15:41:32.753478: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2394560000 Hz
2020-02-03 15:41:32.754356: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55abf7ed7ec0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-03 15:41:32.754417: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
tf.Tensor(-1021.8001, shape=(), dtype=float32)

Per iniciar el docker de tensorflow: 

$ sudo docker run -it --rm tensorflow/tensorflow

________                               _______________                
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ / 
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/


WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...

root@67ef3a0eea06:/#

El python que invoca aquest docker és el python2, i no té compatibilitat GPU (els errors que em dóna). Si vull solucionar aquestes coses he de crear un altre docker: 

$ sudo docker pull tensorflow/tensorflow:latest-gpu-py3

Són 800MB. (cal fer-ho amb sudo??)

I ara iniciem el docker: 

$ sudo docker run -it tensorflow/tensorflow:latest-gpu-py3 bash

$ python --version
Python 3.6.9

$ python -c "import tensorflow as tf;"
2020-02-03 16:02:29.487138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-02-03 16:02:29.904996: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6

$ python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
tf.Tensor(-305.15292, shape=(), dtype=float32)

Però el missatge només surt la primera vegada

Petit tutorial de docker (netejar-ho tot)

per llistar els dockers que tinc: 

$ sudo docker ps -a
$ sudo docker ps -a -q

$ sudo docker stop $(docker ps -a -q)

per aturar: 

$ sudo docker stop 49b9bb83e82e

I per eliminar aquest docker 

$ sudo docker rm 49b9bb83e82e

locate the ID of the images you want to remove: 

$ sudo docker images -a
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
tensorflow/tensorflow   latest-gpu-py3      e2a4af785bdb        3 weeks ago         4.11GB
tensorflow/tensorflow   latest              9bf93bf90865        3 weeks ago         2.47GB
nvidia/cuda             latest              9e47e9dfcb9a        2 months ago        2.83GB
hello-world             latest              fce289e99eb9        13 months ago       1.84kB

i per eliminar les imatges que no ens interessen: 

$ sudo docker rmi IMAGE 9bf93bf90865

per eliminar-ho tot: 

$ sudo docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Si vull eliminar totes les imatges: 

$ sudo docker rmi $(sudo docker images -a -q)

Linux docker postinstall

Tema sudo

The Docker daemon binds to a Unix socket instead of a TCP port. By default that Unix socket is owned by the user root and other users can only access it using sudo. The Docker daemon always runs as the root user.

Per tant, és normal haver d'utilitzar el prefix sudo. Si no ho vull fer, es pot treballar directament com a root, o bé crear el grup docker com es comenta en el tutorial.

Configure Docker to start on boot

Most current Linux distributions (RHEL, CentOS, Fedora, Ubuntu 16.04 and higher) use systemd to manage which services start when the system boots. Ubuntu 14.10 and below use upstart.

A Linux Mint, Debian, Ubuntu, per habilitar o deshabilitar docker en l'inici del sistema: 

$ sudo systemctl enable docker
$ sudo systemctl disable docker

Això és important perquè després de la instal·lació queda habilitat i consumeix recursos, a part de què m'ha donat problemes amb la interfície de xarxa inal·làmbrica.

TensorFlow sense GPU i CUDA

$ sudo docker pull tensorflow/tensorflow:latest-py3

$ sudo docker run -it tensorflow/tensorflow:latest-py3 bash

root@680ddeeaa9d8:/# python --version
Python 3.6.9

root@680ddeeaa9d8:/# python -c "import tensorflow as tf"
2020-02-05 08:41:00.647333: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-05 08:41:00.647773: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-05 08:41:00.647807: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

Continua protestant, però de fet no és un error.

libnvinfer té a verure amb TensorRT (is built on CUDA)

Una possible solució és deshabilitar els warnings: 

root@680ddeeaa9d8:/# python -c "import os;os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2';import tensorflow as tf"

python -c "import os;os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2';import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

tf.Tensor(1021.23694, shape=(), dtype=float32)

0 = all messages are logged (default behavior)
1 = INFO messages are not printed
2 = INFO and WARNING messages are not printed
3 = INFO, WARNING, and ERROR messages are not printed

En la consola de python (>>>) aquests warnings només surten la primera vegada. 

>>> from tensorflow.python.client import device_lib
>>> print(device_lib.list_local_devices())

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 16101204069220344270
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 9804796435611854149
physical_device_desc: "device: XLA_CPU device"
]

Efectivament jo no tinc una GPU que es pugui fer servir. Ara només cal que no vagi donant warnings.

Basic classification: Classify images of clothing

sudo docker run --rm -it \
   --workdir=/app \
   --volume="$PWD":/app \
   --volume="/etc/group:/etc/group:ro" \
   --volume="/etc/passwd:/etc/passwd:ro" \
   --volume="/etc/shadow:/etc/shadow:ro" \
   --volume="/etc/sudoers.d:/etc/sudoers.d:ro" \
   tensorflow/tensorflow:latest-py3 bash

apt-get install joe
joe basic_image_classification_v1.py

basic_image_classification_v1.py: 

from __future__ import absolute_import, division, print_function, unicode_literals

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

# Helper libraries
import numpy as np

print(tf.__version__)

fashion_mnist = keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

train_images.shape
len(train_labels)
train_labels
test_images.shape
len(test_labels)

train_images = train_images / 255.0
test_images = test_images / 255.0

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(train_images, train_labels, epochs=10)

test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)

print('\nTest accuracy:', test_acc)

predictions = model.predict(test_images)
predictions[0]
np.argmax(predictions[0])
test_labels[0]

# python basic_image_classification_v1.py

Quasi ha funcionat fins al final (el meu portàtil no és prou potent i no tinc targeta NVIDIA): 

...
Epoch 10/10
60000/60000 [==============================] - 9s 153us/sample - loss: 0.2382 - accuracy: 0.9115
2020-02-06 13:17:09.182037: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 62720000 exceeds 10% of system memory.
10000/10000 - 1s - loss: 0.3408 - accuracy: 0.8834

Test accuracy: 0.8834
2020-02-06 13:17:10.347075: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 62720000 exceeds 10% of system memory.

$ docker stats

CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
0bdbf375e0c7        determined_feistel   0.00%               1.797MiB / 3.801GiB   0.05%               5.66kB / 0B         639kB / 0B          1

Mentre corre el script es pot veure com el consum de memòria augmenta. Jo no veig que arriba al 10%, però tant se val, la meva màquina no té les característiques per fer aquests càlculs amb soltura. Sembla ser que es pot configurar el docker per acceptar més memòria.

creat per Joan Quintana Compte, febrer 2020

Obtingut de «http://wiki.joanillo.org/index.php?title=TensorFlow._Python&oldid=258901»