What is a Feature in AI?

In machine learning, a feature is an individual measurable property or characteristic of the data that the model uses to learn patterns and make predictions. If you think of a dataset as a spreadsheet, each column is a feature and each row is an example. Features are the raw ingredients that a machine learning model works with.

For example, if you are building a model to predict house prices, your features might include the number of bedrooms, the square footage, the neighborhood, the year the house was built, and the distance to the nearest school. Each of these is a separate piece of information -- a feature -- that the model can use to make its prediction.

Features go by many names in the field. You might hear them called input variables, predictors, attributes, or independent variables. Regardless of the terminology, they all refer to the same thing: the measurable properties that the model receives as input.

The quality and relevance of your features are arguably more important than the choice of algorithm. A simple model with excellent features will almost always outperform a sophisticated model with poor features. This is why experienced data scientists often say: "Garbage in, garbage out." The features you choose fundamentally determine what your model can learn and how well it can learn it.

Feature Types

Features come in several types, and understanding these types is essential because different algorithms handle them differently.

Numerical features are features that represent quantities and can take on continuous or discrete numeric values. Examples include temperature (continuous), age (discrete), salary, height, and test scores. Most machine learning algorithms work naturally with numerical features. They can be further divided into interval features (where the difference between values is meaningful, like temperature) and ratio features (where the ratio is also meaningful, like weight).

Categorical features represent discrete categories or groups without any inherent order. Examples include color (red, blue, green), country of origin, or type of animal. These features need special encoding before they can be used by most algorithms. The most common approach is one-hot encoding, which creates a separate binary column for each category. For example, a "color" feature with three values becomes three columns: is_red, is_blue, and is_green.

Ordinal features are categorical features that do have a meaningful order. Examples include education level (high school, bachelor's, master's, PhD), customer satisfaction rating (1-5), or clothing size (S, M, L, XL). These can be encoded as integers that preserve the ordering, such as 1, 2, 3, 4.

Binary features are the simplest type, representing yes/no or true/false values. Examples include whether a customer is a subscriber (1 or 0), whether an email is spam, or whether a patient has a specific condition. Binary features are already in a format that models can directly use.

Text and image features are unstructured and require significant preprocessing before they can be used as model inputs. Text is typically converted into numerical representations through techniques like TF-IDF, word embeddings, or tokenization. Images are represented as matrices of pixel values, often processed through convolutional layers that automatically extract relevant features.

Feature Engineering

Feature engineering is the art and science of creating new features from existing data to improve model performance. It is often considered the most impactful step in the entire machine learning pipeline and is where domain expertise becomes invaluable.

Consider a dataset of online transactions with a timestamp. The raw timestamp is not very useful on its own, but through feature engineering you can extract the hour of day, day of week, month, whether it is a weekend or holiday, and the time since the customer's last transaction. Each of these derived features captures a different aspect of time-based patterns that the model can learn from.

Common feature engineering techniques include scaling and normalization, which bring all features to a similar range so that no single feature dominates simply because of its scale. Min-max scaling transforms values to a 0-1 range, while standardization transforms them to have zero mean and unit variance. Both are crucial for algorithms like gradient descent that are sensitive to feature scales.

Polynomial features create new features by combining existing ones. If you have features x1 and x2, you can create x1 squared, x2 squared, and x1 times x2. These allow a linear model to capture non-linear relationships. Feature interactions -- products or ratios of existing features -- can reveal patterns that no single feature captures alone.

Binning converts a continuous numerical feature into discrete ranges. Instead of using exact age values, you might create age groups: 0-18, 19-30, 31-50, 51+. This can reduce noise and help the model focus on the broader pattern rather than specific values. Log transformations and square root transformations are used to handle skewed distributions and make features more normally distributed.

The best feature engineers combine deep domain knowledge with creative thinking. A medical researcher building a diagnosis model might create features like "ratio of white blood cells to red blood cells" or "change in blood pressure over last three visits" -- features that capture medical insight no algorithm could discover on its own.

Feature Selection

While feature engineering creates new features, feature selection is the process of choosing which features to keep and which to discard. Not all features are helpful. Some are irrelevant, some are redundant, and some actively hurt model performance by introducing noise or causing overfitting.

There are three main approaches to feature selection. Filter methods evaluate each feature independently of the model using statistical tests. Correlation analysis measures how strongly a feature relates to the target variable. Chi-squared tests work for categorical features. Mutual information quantifies how much knowing one feature tells you about the target. Filter methods are fast and model-agnostic, but they miss feature interactions.

Wrapper methods use the model itself to evaluate feature subsets. The most common approach is recursive feature elimination (RFE), which trains the model, ranks features by importance, removes the least important one, and repeats. Forward selection starts with no features and adds the most useful one at each step. These methods are more accurate but computationally expensive since they retrain the model many times.

Embedded methods perform feature selection as part of the model training process. L1 regularization (Lasso) is the classic example: it drives the weights of irrelevant features to exactly zero, effectively removing them. Decision tree-based models like Random Forest and XGBoost provide built-in feature importance scores that tell you which features contributed most to predictions.

The curse of dimensionality is a key motivation for feature selection. As the number of features grows, the amount of data needed to learn reliable patterns grows exponentially. With too many features and too few examples, the model can find spurious correlations that do not generalize. Reducing the feature space through selection helps the model focus on what matters and avoids this trap.

In practice, feature selection is an iterative process. You try different subsets, evaluate model performance on a validation set, and refine your choices. The goal is the smallest set of features that still delivers strong performance -- a principle aligned with Occam's Razor.

Key Takeaway

Features are the foundation of every machine learning model. They are the individual pieces of information that you feed into the algorithm, and their quality determines the ceiling of what your model can achieve. No amount of algorithmic sophistication can compensate for missing or irrelevant features.

The journey from raw data to model-ready features involves understanding your data types, engineering new features that capture domain knowledge, and selecting the subset that gives the best performance without unnecessary complexity. This process is iterative, creative, and deeply tied to domain expertise.

Whether you are working with tabular data, text, images, or time series, the principle is the same: good features lead to good models. Before reaching for a more complex algorithm, ask yourself whether there are better features you could create from the data you already have. More often than not, the answer is yes, and the improvement from better features will dwarf the improvement from a fancier model.

Remember: features are what the model sees. If the signal is not in the features, the model cannot find it. Invest in your features first, and the rest of the pipeline will follow.