Keras resources

Keras Official docs

Keras is a Python library for deep learning that runs on TensorFlow, Theano or CNTK.

Keras is the ideal library for rapid experimentation. Currently, the biggest downside to Keras is that it doesn’t support multi-GPU environments for parallel training (see this discussion).

Keras was developed and maintained by François Chollet, a Google engineer using four guiding principles:

  • Modularity: A model can be understood as a sequence or a graph alone. All the concerns of a deep learning model are discrete components that can be combined in arbitrary ways.
  • Minimalism: The library provides just enough to achieve an outcome, no frills and maximizing readability.
  • Extensibility: New components are intentionally easy to add and use within the framework, intended for researchers to trial and explore new ideas.
  • Python: No separate model files with custom file formats. Everything is native Python.

Keras examples  – General & Basics

In this post, you discovered how to create your first neural network model using the powerful Keras Python library for deep learning.

Specifically, you learned the five key steps in using Keras to create a neural network or deep learning model, step-by-step including:

  1. How to load data.
  2. How to define neural network in Keras.
  3. How to compile a Keras model using the efficient numerical backend.
  4. How to train a model on data.
  5. How to evaluate a model on data.

Tips for Hyperparameter Optimization

Below are some handy tips to consider when tuning hyperparameters of your neural network.

  • k-fold Cross Validation. You can see that the results from the examples in this post show some variance. A default cross-validation of 3 was used, but perhaps k=5 or k=10 would be more stable. Carefully choose your cross validation configuration to ensure your results are stable.
  • Review the Whole Grid. Do not just focus on the best result, review the whole grid of results and look for trends to support configuration decisions.
  • Parallelize. Use all your cores if you can, neural networks are slow to train and we often want to try a lot of different parameters. Consider spinning up a lot of AWS instances.
  • Use a Sample of Your Dataset. Because networks are slow to train, try training them on a smaller sample of your training dataset, just to get an idea of general directions of parameters rather than optimal configurations.
  • Start with Coarse Grids. Start with coarse-grained grids and zoom into finer grained grids once you can narrow the scope.
  • Do not Transfer Results. Results are generally problem specific. Try to avoid favorite configurations on each new problem that you see. It is unlikely that optimal results you discover on one problem will transfer to your next project. Instead look for broader trends like number of layers or relationships between parameters.
  • Reproducibility is a Problem. Although we set the seed for the random number generator in NumPy, the results are not 100% reproducible. There is more to reproducibility when grid searching wrapped Keras models than is presented in this post.

Keras examples  – Images

Keras examples – Text mining & NLP

After reading a few papers about NLP, and specifically deep learning applications, the author decided to go ahead and try out a few things on my own. In this post the author will demonstrate a character level models for sentiment classification. The models are built with the author’s favourite framework Keras (with Tensorflow as back-end). In case you haven’t used Keras before I strongly suggest it, it is simple and allows for very fast prototyping (thanks François Chollet. After version 1.0 and the introduction of the new functional API, creating complex models can be as easy as a few lines. I’m hoping to demonstrate some of it’s potential as we go along.

If you want to get familiar with the framework the author would suggest following these links:

 Assume that most people reading this have some basic knowledge about convolution networks, mlps, and rnn/lstm models.

A very popular post from Andrej Karpathy talking about the effectiveness of recurrent nets presents a character level language model built with RNNs, find it here. A simple implementation of this model for Keras, here.

======GPU and distributed (Keras + TensorFlow)

  • Distributed deep learning with Keras and Apache Spark (GitHub repo)

Spark 2.0

If you want to run the examples using Apache Spark 2.0.0 and higher. You will need to remove the line containing sqlContext = SQLContext(sc). We need to do this because in Spark 2.0+, the SQLContext, and Hive context are now merged in the Spark session.