Fully Convolutional Networks(FCN) for Semantic Segmentation
In this post, I will implement Fully Convolutional Networks(FCN) for semantic segmentation on MIT Scence Parsing data.
While object detection methods like R-CNN heavily hinge on sliding windows (except for YOLO),
FCN doesn’t require it and applied smart way of pixel-wise classification. To make it possible, FCN implements 3 distinctive features in their network.
Get Spatial map (heatmap) instead of non-spatial outpues by chaning Fully Connected Layer into 1 X 1 Convolution
In-network Upsampling for pixel-wise prediction
Combine layers of the feature hierarchy to get local predictions with respect to global strucutre, also called as deep-jet
Details of Features
With regard to “1. Spatial Map”, it allows us to get heatmap not vector output, which convert classification problem into coarse spatial map.
Upsampling the coarse spatial map into pixel of original image is performed through in-network nonlinear upsampling (DeConvolution). Since DeConvolutioon is implemented using reverse the forward and backward of convolution, the filter(e.g. ‘W_t1’) of DeConvoltion layer(‘tf.nn.conv2d_transpose’ in Tensorflow) can also be learned.
Finally, upsampling 32X generates srikingly coarse ouptuts. To overcome such phenomena, the author added the 2X upsample of last layer and the right before layer. This combination of layers of feature hierarchy expands the capabililty of network to make fine local prediction with respcet to global structures. The “fuse” in the code of figure corresponds to the combination of layers, and the combination procedure was performed up to 2 times that make us 8X upsamlple. Therefore, FCN-32s is converted to FCN-8s via Skip Architecture.
To train and validate the FCN-8s in Windows, I modified two scripts: read_MITSceneParsingData.py & FCN.py.
First and foremost, the script read_MITSceneParsingData.py evokes an issue of parsing, which disables getting training and validation data.
Last but not least, I changed the batch size from 2 to 10, which significantly reduced cost and ‘main’ function for validation (NVIDIA GPU Geforce 1080ti 11G is used).
One of random validation samples is listed below. The quality of prediction result is not precise as much as I expected. One possible reason is the number of classes in read_MITSceneParsing data (151), while the paper used PASCAL VOC 2011 data (21).
Reulsts in Paper
Resources that I referenced are listed at the bottom of this post.
AAE is better than VAE in terms of Reconstruction error and Diversity
Unique Scaffolds can be achieved
Considering 2D strucutre is critical
3D similarity based on ligands
To be compact, I didn’t upload Input, related property, and data resources though being created.
If somebody pursue for more details about each paper, please feel free to email me.
For the sake of compactness, I would demonstrate two methods Hillclimb MLE (HC-MLE) in Reinfocement Learning and Monte Carlo Tree Search from BenovolentAI and ChemTS papers. The second paper, VAE with Property, is reviewed in my previous post.
1. Hillclimb MLE (HC-MLE)
First, There are 19 benchmarks that used for Reward in Reinforcement Learning. They can be catagorized into Validity, Diversity, Physio-Chemical Property, similarity with 10 representative compounds, Rule of 5, and MPO.
Secondly, HC-MLE mazimizes the likelihood of sequences that received Top K highest reward.
2. Monte Carlo Tree Search (MCTS)
Pretrain RNN and Get Conditional Probability
Conditional Probability is used in MCTS as sampling distribution to get next character and elongate the smiles code
Reward score of generated smiles code is computed
In Back propagation, reward is back propagated & UCB at each node is updated
benevolent (EXPLORING DEEP RECURRENT MODELS WITH REINFORCEMENT LEARNING FOR MOLECULE DESIGN): benevolent
VAE with Property (Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules): vae_property
druGAN (druGAN: An Advanced Generative Adversarial Autoencoder Model 2 for de Novo Generation of New Molecules with Desired Molecular 3 Properties in Silico ): druGAN
InSilico (In silico generation of novel, drug-like chemical matter using the LSTM neural network ): InSilico
DeepChem (Applying Automated Machine Learning to Drug Discovery) : N/A
SwissTarget (SwissTargetPrediction: a web server for target prediction of bioactive small molecules ): SwissTarget
De Novo Design: Derivatives of Autoencoder
A primary goal of designing a deep learning architecture is to restrict the set of functions that can be learned to ones that match the **desired properties** from the domain
S Kearnes etal 2016 (Molecular Grpah Convolutions: Moving Beyond Fingerprints)
When building deep learning architecture, the above statement strongly highlights the key concept of implementing deep learning architecture. Especially, phrase “match the desired properties” can be understood as teaching, imposing a constraint, or regularization in several learning algorithm. For instance, GAN attempts to impose regularization by introducing Discriminator & Generator on the purpose of setting, and derivatives of Autoencoder predict a certain property by attaching MLP from latent space to get new datum with desired properties. Following takeaway outlines what I will illustrate in series with respect to generative models.
Structure of Generative Models
As Ian Goodfellow (NIPS 2014, and 2016 tutorial) demonstrates that maximizing Maximum Likelihood Estimates is equivalent to minimizing KL divergence between data generating distribution and the model and introduces Nash equilibrium, which verifies the unbiasness of GAN. On top of that, Generating a datum via Variational Autoencoder while simultaneously predicing property can be regarded as optimization with constraint. whether explicitily represent a probability distribution or not derives the fundamental difference between these two generative models(GAN & Variational Autoencoder).
Although the way of approach is different, convergence of disciplines such as Economcis, Industrial Engineering, Statistics, and Computer Science has appeared and they share the goal of implementing natural “regularization”.
As for the widley known argloithm, foramts of regularization can be categorized into 2 folds: Game(Adversarial) and Predict property simultaneously(Constraint). By starting from the perspective of Autoencoder, I made 3 overlapping diagrams where “Adversarial” and “Constraint” lie on the domain of “regularization”.
1. Comparison of Models
Below picture is combination of figures from different papers. Red box indicated in derivatives of autoencoder denotes different substructure across derivatives.
Distinctive Features across Methods
AE –> VAE
Reparametrization from encoder to Latent Space via pre-assumed distribution (usually Gaussian distribtion)
VAE –> VAE with Property
Added a netwrok from latent space to a supervised network such as Regression or Classification, which agglomerates Latent Space according to the level of target value.
AE –> AAE
No Reparametrization, but adversarial Network is attached
Cost function across Methods
AE : Reconstruction error
VAE : Reconstruction error + KL-divergence (Regularizing Variational Parameter)
( latent space is driven by adding stochasticity to the encoder ) –> Adding noise to the encoded latent space forces the decoder to learn a wider range of latent points, which lead to find out more robust representations.
VAE with property: Reconstruction error of decoder + KL-divergence (Regularizing Variational Parameter) + regression error
AAE : Reconstruction error of decoder + Discriminator loss + encoder(Generator) loss
In terms of regularizing the encoder posterior to match a pre-assumed prior, VAE with property and Adversarial autoencoder are on the same domain.
Specifically, Discriminator and Generator loss can be implemented like following
Discriminator related loss:
Generator related loss:
2. Data & Pre-processing
Image or sequence (Molecular information is denoted as SMILES code, which is a sequence, will be mainly used )
In this post, I will use data from Tox21 data and the input will be SMILES code, which is mainly used for chemical design and molecular property prediction. Data pre-processing and specific model illustration is described below ( reference: T Blaschke etal 2017 ).
To adopt periodic table into SMILES code, I merged chemical in periodic table with the code and generated one-hot encoded matrix for each SMILES code. Since the max length of chemical is 2, ismol variable in the function is
3. Implementation of VAE with Proprty
3.1 HyperParameter Setting
3.5 VAE Loss
Kearnes, Steven, et al. “Molecular graph convolutions: moving beyond fingerprints.” Journal of computer-aided molecular design 30.8 (2016): 595-608.
Blaschke, Thomas, et al. “Application of generative autoencoder in de novo molecular design.” Molecular informatics 37.1-2 (2018).
Gómez-Bombarelli, Rafael, et al. “Automatic chemical design using a data-driven continuous representation of molecules.” ACS Central Science 4.2 (2018): 268-276.
Goodfellow, Ian, et al. “Generative adversarial nets.” Advances in neural information processing systems. 2014.
The task that I encountered was to construct a program, which can run by itself and should keep watching multiple Database tables at different time intervals. To make it possible, I used the thread with TCP/IP to check whether new data tables are updated in SQL. If new data is found, then I access the data while remembering previously processed one. After transforming the newly scanned data into a 2D matrix and projecting it as an image, I implemented trained CNN model to predict the grade of a product. Finally, I write the prediction result with corresponding information on an outcome table and saved the image into the path(written in JSON file).
Structure of Codes
Global Variables: Required information to access SQL
JSON file: Paths where to save image and where trained model parameter located
Class A: SQL manager
Class B: Deep Learning Model and Pre-Processing
Function C: scheduler, which calls the functions in Class A and Class B to check whether a new data came in and execute to save the image and perform prediction.
3. Main function
Class A initialization enables us to connect database.
Class B initialization read JSON file for necessary paths
SetModel function in Class B ensures whether trained model parameters are loaded
Catch for Function C
Build executable file(.exe) using pyinstaller
Things to remember
If Tensorflow environment is CPU(GPU) based, trained model should be built upon CPU(GPU)
Type in required packages when building an executable file
Example of generating one python executable file in command line.
When I encountered to work on multiple projects at the same time, I was supported for using a workstation with 4 GPUs(GeForce 1080Ti, 11G). While delving in to find out the way to efficiently utilize given resources, I figured out that Tensorflow offers some options that enable us to assign multiple GPUs. Below contents describe how to assign GPUs corresponding to your intention, tasks, and models. Specifically, there could be 3 cases. First, multiple jobs(scripts) at different GPUs. Secondly, multiple models at each GPU. Finally, one big model to distributed GPUs.
1. Multi tasks –> Multi GPUs
When you have multiple projects(scripts) and each project requires to use GPU, you can assign each GPU to the corresponding project(script). Without declaration about the list of the visible device, Tensorflow captures all accessible GPUs. To prevent such holdings, assigning Configuration with GPU naming in Session is necessary. Suppose that model is already declared and training with specified epoch will be followed.
2. Multi models –> Multi GPUs
If implementing multiple models at each GPUs(e.g. ensemble) is required, each GPU can be allocated to models(graph). Additionally, if each graph shares lots of common parts such as input, output, dropout, and cost, declare graph with “scope” allow us to access each component of graph more efficiently at training parts. In order to show how to access shared component name across the different graph, I only changed the size of the node at the last layer of each graph and named components same.
Although runtime might vary depending on the size of batch and data, training graph parallel across GPUs shorten the time about 1 sec for each iteration rather than training each graph sequentially(e.g. MNIST data with 10 batch size is used).
3. One big model –> Distributed GPUs
Once the size of necessary model exceeds that of GPU memory, the model can be divided into multiple jobs where each of them are executed by different GPUs.
Hopefully, I wish to improve the performance of carrying out the code by properly applying described methods with multi-GPU.
On the other hand, although GPU seems to be powerful for computation-intensive works, it might slow down for the case of running conditional statements relative to CPU. Sp, optimally assigning each task or job to appropriate memory can be another option to boost the performance of work.