Neural Architecture Search: Automating Network Design

Designing neural network architectures has traditionally been a craft, requiring deep expertise and extensive experimentation. Researchers spend months tweaking layer configurations, skip connections, and activation functions to find architectures that work well. Neural Architecture Search (NAS) aims to automate this process, using algorithms to explore the vast space of possible architectures and discover designs that match or exceed human-crafted ones.

The NAS Framework

Every NAS method consists of three components:

• Search space: Defines the set of possible architectures. This includes which operations are available (convolution, pooling, skip connections), how they can be connected, and the overall structure (e.g., cells that are stacked to form the full network).
• Search strategy: The algorithm used to explore the search space. Options include reinforcement learning, evolutionary algorithms, Bayesian optimization, and gradient-based methods.
• Performance estimation: How candidate architectures are evaluated. Training each architecture from scratch to convergence is prohibitively expensive, so various proxy methods are used.

Search Strategies

Reinforcement Learning

Google Brain's seminal NAS paper (Zoph and Le, 2017) used a recurrent neural network as a controller that generated architecture descriptions. The controller was trained with reinforcement learning, using the validation accuracy of the generated architecture as the reward signal.

This approach was effective -- NASNet, discovered this way, achieved state-of-the-art results on ImageNet -- but it was extraordinarily expensive. The original search required 800 GPUs running for 28 days, costing millions of dollars in compute.

Evolutionary Methods

Evolutionary approaches maintain a population of architectures. In each generation, top performers are selected as parents, offspring are created through mutations (adding/removing layers, changing operations), and underperformers are eliminated. AmoebaNet, found through evolutionary search, matched NASNet's accuracy.

Evolutionary methods are naturally parallelizable and do not require differentiable objectives, making them flexible. However, they still require evaluating many candidate architectures.

Early NAS methods proved that algorithms could design neural networks as well as human experts, but at computational costs that only the largest tech companies could afford.

Differentiable NAS (DARTS)

DARTS (Differentiable Architecture Search), proposed by Liu et al. in 2019, made NAS dramatically more efficient. Instead of treating architecture search as a discrete optimization problem, DARTS relaxes the search space to be continuous.

In DARTS, each edge in the architecture graph is a weighted mixture of all possible operations. These mixture weights (architecture parameters) are optimized jointly with the network weights using gradient descent. After search, the operation with the highest weight on each edge is selected, yielding the final discrete architecture.

DARTS reduced the search cost from thousands of GPU-days to a single GPU-day, making NAS accessible to researchers without massive compute budgets.

Notable NAS-Discovered Architectures

EfficientNet

Perhaps the most commercially impactful NAS result, EfficientNet (Tan and Le, 2019) used NAS to find a baseline architecture (EfficientNet-B0) and then introduced compound scaling -- a principled way to scale width, depth, and resolution simultaneously. The EfficientNet family achieved better accuracy than previous models at a fraction of the computational cost.

MobileNetV3

Designed for mobile and edge deployment, MobileNetV3 combined NAS with platform-aware optimization. The search explicitly accounted for latency on mobile hardware, finding architectures that were fast on real devices rather than just theoretically efficient.

Once-for-All Networks

MIT's Once-for-All (OFA) approach trains a single large network from which many sub-networks can be extracted without retraining. NAS is then used to find the optimal sub-network for each target platform, enabling deployment across devices with vastly different compute capabilities from a single training run.

Challenges and Limitations

Despite its promise, NAS faces several ongoing challenges:

• Search space design: The human-designed search space heavily influences what NAS can find. Poor search space design limits NAS to incremental improvements over existing architectures.
• Reproducibility: DARTS and similar methods can be sensitive to hyperparameters and random seeds, with different runs finding different architectures.
• Performance collapse: DARTS sometimes converges to degenerate architectures dominated by skip connections, requiring regularization to prevent.
• Transfer across tasks: Architectures found for one task or dataset do not always transfer well to others.
• Scale: Most NAS research operates on small proxy tasks (CIFAR-10) and it remains expensive to search directly on large-scale problems.

NAS Today and Tomorrow

The NAS field has matured considerably. Modern approaches like zero-cost proxies estimate architecture quality from random weights without any training, reducing search costs to minutes. Hardware-aware NAS co-optimizes architecture and hardware mapping for deployment on specific accelerators.

Interestingly, the rise of transformers and LLMs has shifted the focus away from architecture search in some areas. When the transformer architecture works well across nearly all tasks, the architecture search problem becomes one of scaling and configuration rather than fundamental topology design.

However, NAS remains relevant for edge AI, where hardware constraints make architecture efficiency critical, and for specialized domains like medical imaging and autonomous driving, where task-specific architectures can provide meaningful advantages over general-purpose designs.