{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# Deferred Initialization\n",
    ":label:`sec_deferred_init`\n",
    "\n",
    "So far, it might seem that we got away\n",
    "with being sloppy in setting up our networks.\n",
    "Specifically, we did the following unintuitive things,\n",
    "which might not seem like they should work:\n",
    "\n",
    "* We defined the network architectures\n",
    "  without specifying the input dimensionality.\n",
    "* We added layers without specifying\n",
    "  the output dimension of the previous layer.\n",
    "* We even \"initialized\" these parameters\n",
    "  before providing enough information to determine\n",
    "  how many parameters our models should contain.\n",
    "\n",
    "You might be surprised that our code runs at all.\n",
    "After all, there is no way the deep learning framework\n",
    "could tell what the input dimensionality of a network would be.\n",
    "The trick here is that the framework *defers initialization*,\n",
    "waiting until the first time we pass data through the model,\n",
    "to infer the sizes of each layer on the fly.\n",
    "\n",
    "\n",
    "Later on, when working with convolutional neural networks,\n",
    "this technique will become even more convenient\n",
    "since the input dimensionality\n",
    "(i.e., the resolution of an image)\n",
    "will affect the dimensionality\n",
    "of each subsequent layer.\n",
    "Hence, the ability to set parameters\n",
    "without the need to know,\n",
    "at the time of writing the code,\n",
    "what the dimensionality is\n",
    "can greatly simplify the task of specifying\n",
    "and subsequently modifying our models.\n",
    "Next, we go deeper into the mechanics of initialization.\n",
    "\n",
    "\n",
    "## Instantiating a Network\n",
    "\n",
    "To begin, let us instantiate an MLP.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "origin_pos": 2,
    "tab": [
     "tensorflow"
    ]
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "net = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Dense(256, activation=tf.nn.relu),\n",
    "    tf.keras.layers.Dense(10),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 3
   },
   "source": [
    "At this point, the network cannot possibly know\n",
    "the dimensions of the input layer's weights\n",
    "because the input dimension remains unknown.\n",
    "Consequently the framework has not yet initialized any parameters.\n",
    "We confirm by attempting to access the parameters below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "origin_pos": 5,
    "tab": [
     "tensorflow"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[], []]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[net.layers[i].get_weights() for i in range(len(net.layers))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 7,
    "tab": [
     "tensorflow"
    ]
   },
   "source": [
    "Note that each layer objects exist but the weights are empty.\n",
    "Using `net.get_weights()` would throw an error since the weights\n",
    "have not been initialized yet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 10
   },
   "source": [
    "Next let us pass data through the network\n",
    "to make the framework finally initialize parameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "origin_pos": 12,
    "tab": [
     "tensorflow"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(20, 256), (256,), (256, 10), (10,)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = tf.random.uniform((2, 20))\n",
    "net(X)\n",
    "[w.shape for w in net.get_weights()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 13
   },
   "source": [
    "As soon as we know the input dimensionality,\n",
    "20,\n",
    "the framework can identify the shape of the first layer's weight matrix by plugging in the value of 20.\n",
    "Having recognized the first layer's shape, the framework proceeds\n",
    "to the second layer,\n",
    "and so on through the computational graph\n",
    "until all shapes are known.\n",
    "Note that in this case,\n",
    "only the first layer requires deferred initialization,\n",
    "but the framework initializes sequentially.\n",
    "Once all parameter shapes are known,\n",
    "the framework can finally initialize the parameters.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Deferred initialization can be convenient, allowing the framework to infer parameter shapes automatically, making it easy to modify architectures and eliminating one common source of errors.\n",
    "* We can pass data through the model to make the framework finally initialize parameters.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. What happens if you specify the input dimensions to the first layer but not to subsequent layers? Do you get immediate initialization?\n",
    "1. What happens if you specify mismatching dimensions?\n",
    "1. What would you need to do if you have input of varying dimensionality? Hint: look at the parameter tying.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 15,
    "tab": [
     "tensorflow"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/281)\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}