This resource contains help functions/steps for following tasks:
- Using ColumnTransformer here
- Search for available GPUs
- Steps to handle nan in loss during training
- Steps to increase gpu utilization and speed of processing
- Commands to monitor gpu usage
- Commands to run tensorboard
- Command to create tar file
- Command to compute per user memory usage
- Shell script to compress all files in a directory and delete the originial file from the directory. This script will create separate tar file for each file in the directory
- Setting up tensorflow environment variables
- Setting up LD_LIBRARY_PATH variable for tensorflow
- Some useful steps to setup server for Deep Learning
- Shell command to check folder wise disk usage
- Filezilla: Error: Disconnected: No supported authentication methods available (server sent: publickey)
- How to add equations to you Readme file in github?
- Important links for Jekyll
- How to create personal access token for git?
- How to attach virtual environment to jupyter notebook?
- VS Code Anaconda Environment - import _ssl DLL load fail error
Links
S.No. | Category | Task | Link |
---|---|---|---|
1. | Tensorflow | Tensorflow version compatibility chart | https://www.tensorflow.org/install/source |
2. | Cuda, Tensorflow | Steps needed to install cuda & runtime libraries | https://www.tensorflow.org/install/gpu#ubuntu_1604_cuda_10 |
Using ColumnTransformer This code demonstrates how to use ColumnTransformer
from sklearn.compose import ColumnTransformer
from sklearn.compose import make_column_transformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
df = pd.DataFrame({'region':['a','a','b','c'], 'country':['c1', 'c1', 'c2', 'c3']})
categorical_features = ['region','country']
preprocessor = make_column_transformer((OneHotEncoder(), categorical_features))
df_new = preprocessor.fit_transform(df)
Search for available GPUs How to get list of GPUs visible to tensorflow/keras
from keras import backend
backend.tensorflow_backend._get_available_gpus()
import keras
import tensorflow as tf
config = tf.ConfigProto( device_count = {'GPU': 1 , 'CPU': 56} )
sess = tf.Session(config=config)
keras.backend.set_session(sess)
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
import tensorflow as tf
tf.test.is_gpu_available() tells if the gpu is available
tf.test.gpu_device_name returns the name of the gpu device
with tf.Session() as sess:
devices = sess.list_devices()
Steps to handle nan in loss during training:
- Normalize your outputs by quantile normalizing or z scoring. To be rigorous, compute this transformation on the training data, not on the entire dataset.
- Add regularization, either by increasing the dropout rate or adding L1 and L2 penalties to the weights.
- reduce the size of your network.
- Increase the batch size from 32 to 128. 128 is fairly standard and could potentially increase the stability of the optimization.
- In Keras you can use clipnorm=1 (see https://keras.io/optimizers/) to simply clip all gradients with a norm above 1.
- It turns out, one of the images that I was handing to my CNN (and doing mean normalization on) was nothing but 0’s. I wasn’t checking for this case when I subtracted the mean and normalized by the std deviation and thus I ended up with an exemplar matrix which was nothing but nan’s.
- The first thing you can try is changing your activation to LeakyReLU instead of using Relu or Tanh.
- Column in my data set had all the same numerical value, making it effectively a worthless addition to the DNN.
Steps to increase gpu utilization and speed of processing
- Clear Keras session using K.clear_session
- Use fit_gen instead of fit if data is huge
- Use early stopping
- Use multiple workers in fit/fit_gen. (This has no been checked)
- Try increasing validation_freq
- If using LSTM try using cuDNNLSTM instead of LSTM
- If using LSTM try using unroll = True option
Commands to monitor gpu usage $ nvidia-smi -l 1 $ watch -n 0.5 nvidia-smi
Commands to run tensorboard tensorboard –logdir=./iteration_7/model/logs/ –port=8097 –host=127.0.0.1
Command to create tar file tar -czvf moscow.tar.gz ./store_log_calendar_Moscow.csv
Command to compute per user memory usage ps aux | awk ‘{arr[$1]+=$4}; END {for (i in arr) {print i,arr[i]}}’ | sort -k2
Shell script to compress all files in a directory and delete the originial file from the directory. This script will create separate tar file for each file in the directory for i in * ; do tar cvzf $i.tar.gz $i; rm -rf $i; done
Setting up tensorflow environment
Environment variables for tensorflow-gpu: TF_FORCE_GPU_ALLOW_GROWTH = True
Configuration parameters to be set in code
config.gpu_options.allow_growth = True
config.gpu_options.visible_device_list = which_gpu
config.gpu_options.per_process_gpu_memory_fraction = 0.4
Code to dynamically grow GPU memory (instead of allocating everything at once)
import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True # dynamically grow the memory used on the GPU
sess = tf.Session(config=config)
set_session(sess)
Some useful steps to setup server for Deep Learning
Download the anaconda installer $ wget https://repo.anaconda.com/archive/Anaconda3-2018.12-Linux-x86_64.sh or $ wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh
Run the installer $ bash Anaconda3-2019.03-Linux-x86_64.sh $ source .bashrc
Upgrade conda (Might not be required, check the instructions) conda upgrade conda or conda upgrade –all
Create new virtual environment in conda $ conda create -n venv_rb python=3.5 $ conda activate venv_rb
Some useful commands to check the environment $ conda info $ conda list $ which pip
Installing some useful packages $ pip install –upgrade tensorflow-gpu $ pip install keras $ conda install pandas $ pip install matplotlib $ pip install -U scikit-learn $ pip install jupyter $ pip install pandas-profiling
Setting up LD_LIBRARY_PATH variable for tensorflow export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64
Additional steps needed to install cuda & runtime libraries (Select based on ubuntu & tensorflow-gpu versions) https://www.tensorflow.org/install/gpu#ubuntu_1604_cuda_10
Tensorflow version compatibility chart https://www.tensorflow.org/install/source
Shell command to check folder wise disk usage $ du -h –max-depth=1 /data/rb/users/|sort -h
Filezilla error while connecting
Error: Disconnected: No supported authentication methods available (server sent: publickey) Error: Could not connect to server
Solution If you have private key file (.ppk file): In Filezilla, Go to Edit » Settings » SFTP » Add Key File Upload your private key file and try to re-connect.
How to add equations to you Readme file in github
Solution https://stackoverflow.com/questions/35498525/latex-rendering-in-readme-md-on-github Take your latex equation and go to http://www.codecogs.com/latex/eqneditor.php, at the bottom of the area where your equation appears displayed there is a tiny dropdown menu, pick URL encoded and then paste that in your github markdown
Important links for jekyll
Solution
- Install jekyll on windows
- Link to jekyll themes
- Ruby installers
- How to apply new theme to your jekyll website
- Set the name of the theme in _config.yml
theme: jekyll-theme-cayman plugins: - jekyll-feed
- Set the name of the theme in Gemfile
gem "jekyll-theme-cayman"
- Set the name of the theme in _config.yml
- How to iterate over posts in _posts directory of your website
How to create personal access token for git
How to attach virtual environment to jupyter notebook?
Following commands have been tested on Windows 10. Open Anaconda prompt and run the following commands
Update conda
$ conda update -n base -c defaults conda
Switch to your virtual environment using
$ conda activate venv
Install jupyter, ipykernel and attach
$ pip install jupyter
$ pip install ipykernel
$ python -m ipykernel install --user --name=venv
Installed kernelspec venv in C:\Users\<<username>>\AppData\Roaming\jupyter\kernels\venv
In this folder you can find Kernel.json
{
"argv": [
"C:\\Users\\<<username>>\\anaconda3\\envs\\venv\\python.exe",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "venv",
"language": "python",
"metadata": {
"debugger": true
}
}
Now, you should be able to see this environment venv in your jupyter list of kernels
VS Code Anaconda environment - import _ssl DLL load fail error
There are couple of solutions mentioned on stackoverflow here and in the above link. What worked for me :
Open Anaconda Prompt
Activate your virtual environment
conda activate <<your virtual environment>>
Run vscode from this prompt
(venv) C:\Users\nitagarwal> code