DEV Community

The Machine Learning Lifecycle: 10 Steps From Problem to Production (And Why Most Projects Fail at Step 3)

Ege Pakten — Sat, 25 Apr 2026 09:15:35 +0000

Every ML tutorial jumps straight to model training. But in the real world, training is step 7 out of 10 — and the steps before it are where projects succeed or fail. This post walks through the full Machine Learning Lifecycle, from defining your problem to keeping your model healthy in production, with real examples and practical advice at every stage.

The Big Picture

Machine Learning is an iterative and structured process. It's not "throw data at an algorithm and hope for magic." It's a cycle — and most teams loop through it multiple times before they get something that works in production.

Here are the 10 stages:

1. Problem Definition → 2. Data Collection → 3. Data Cleaning & Preprocessing
→ 4. Exploratory Data Analysis (EDA) → 5. Feature Engineering & Selection
→ 6. Model Selection → 7. Model Training → 8. Model Evaluation & Tuning
→ 9. Model Deployment → 10. Monitoring & Maintenance → (back to 1)

Let's go through each one.

1. Problem Definition — "What are we actually solving?"

This is where most failed ML projects go wrong. Before touching any data or code, you need to answer:

What business problem am I solving? Not "I want to use AI," but "I want to reduce customer churn by 15%" or "I want to detect fraudulent transactions in real time."
Is ML even the right tool? Sometimes a simple rule-based system or a SQL query is better. ML is expensive overkill for problems that have clear, deterministic rules.
What does success look like? Define a measurable metric: accuracy, precision, recall, revenue impact, latency requirements.
What type of ML problem is this? Classification? Regression? Clustering? Recommendation? This dictates everything downstream.

The problem definition dictates the type of data you need. If you define the problem wrong, you'll collect the wrong data, build the wrong model, and ship something nobody wanted.

Example: A bank wants to "use AI." That's not a problem definition. "Predict which credit card transactions are fraudulent with less than 0.1% false positive rate and under 200ms latency" — that's a problem definition.

2. Data Collection — "Do we have enough?"

Once you know what you're solving, you need data. This step is about gathering enough high-quality, relevant data to train a model.

Key questions:

Where does the data come from? Internal databases, APIs, web scraping, third-party vendors, public datasets, user-generated content?
How much data do I need? Depends on complexity. A simple classifier might need 1,000 examples. A computer vision model might need 100,000+ labeled images. An LLM needs billions of tokens.
Is the data labeled? For supervised learning, you need labels (the "right answers"). Labeling is often the most expensive and time-consuming part.
Is the data representative? If you train a facial recognition system only on photos of one demographic, it will fail on others. Your data must represent the real-world distribution.

Common pitfalls:

Assuming you have "big data" when you actually have big noise
Not checking for sampling bias
Ignoring data privacy regulations (GDPR, KVKK, HIPAA)
Collecting too many features and not enough samples

3. Data Cleaning and Preprocessing — "Garbage in, garbage out"

This is where you spend 60-80% of your actual project time. Raw data is messy. Always.

What you're doing here:

Handling Missing Values

Some rows have blank fields. Do you fill them with the mean? The median? A prediction? Or drop them entirely?
The right answer depends on why the data is missing. "Random missing" and "systematically missing" require different approaches.

Removing Duplicates

Duplicate records distort your model's understanding of the distribution.

Fixing Inconsistent Data

"New York", "new york", "NY", "N.Y." are the same city but four different strings.
Date formats: "04/20/2026" vs "2026-04-20" vs "20 April 2026"
Units: meters vs feet, Celsius vs Fahrenheit

Handling Outliers

A salary dataset where most values are $40K-$120K but one entry says $99,999,999. Is it real or a typo? Outliers can destroy model performance or provide critical signal — you have to decide.

Data Type Conversions

Categorical variables need encoding (one-hot, label encoding)
Text needs tokenization
Images need resizing, normalization
Dates need feature extraction (day of week, month, holiday flag)

Normalization and Scaling

Features on different scales (age: 0-100, salary: 20,000-500,000) can bias models that use distance calculations. Standard scaling (z-score) or min-max scaling fixes this.

The motto: garbage in, garbage out. No model, no matter how sophisticated, can learn good patterns from bad data.

4. Exploratory Data Analysis (EDA) — "What does the data actually look like?"

Before building any model, you need to understand your data. EDA is about getting the big picture.

What you're looking for:

Distributions — Is your target variable balanced? If 99% of transactions are legitimate and 1% are fraud, you have a class imbalance problem.
Correlations — Which features are related to each other? Which features predict your target?
Patterns and trends — Seasonal effects? Time-based shifts? Geographic clusters?
Data quality issues you missed in step 3 — Sometimes problems only become visible in visualization.

Tools: histograms, scatter plots, correlation matrices, box plots, pair plots. Libraries: Pandas, Matplotlib, Seaborn, Plotly.

Example: You're building a house price predictor. EDA reveals that "number of bedrooms" and "square footage" are highly correlated (0.92). Including both might cause multicollinearity. You might drop one or combine them.

EDA often sends you back to step 2 or 3 — you realize you need more data, or your data has problems you didn't see before.

5. Feature Engineering and Selection — "Good features > fancy models"

This is often the difference between a mediocre model and a great one. Feature engineering is the art of creating new input variables that help the model learn patterns better.

Feature Engineering (Creating)

From dates: extract day_of_week, is_weekend, month, quarter, days_since_last_event
From text: word count, sentiment score, TF-IDF values, embeddings
From location: distance to nearest city, population density, latitude buckets
Combining features: price_per_sqft = price / square_footage, BMI = weight / height²
Domain knowledge: a doctor knows that "blood pressure × age" interaction matters; encode that

Feature Selection (Removing)

Not all features help. Some add noise. Too many features cause overfitting and slow training. Techniques:

Correlation analysis — drop features that are highly correlated with each other
Feature importance from tree-based models (Random Forest, XGBoost)
Recursive Feature Elimination (RFE) — iteratively remove the least important feature
L1 Regularization (Lasso) — automatically zeroes out unimportant features during training

Key insight: a simple model with great features almost always beats a complex model with bad features.

6. Model Selection — "Choose the right tool for the job"

Now you pick which algorithm(s) to try. This depends on:

Problem type: Classification → Logistic Regression, Random Forest, SVM, Neural Network. Regression → Linear Regression, XGBoost, Neural Network. Clustering → K-Means, DBSCAN. Sequence → RNN, LSTM, Transformer.
Data size: Small data → simpler models (logistic regression, SVM). Large data → deep learning can shine.
Interpretability needs: Healthcare and finance often need explainable models (decision trees, linear models). Recommendation engines can afford black boxes (deep learning).
Latency requirements: Real-time inference needs fast models. Batch processing can afford slower ones.

Best practice: Start simple. Try logistic regression or a decision tree first. If it gets 85% accuracy, you have a strong baseline. Then try more complex models and see if the improvement justifies the complexity.

You often try 3-5 different models and compare their performance.

7. Model Training — "The model learns about the data"

This is the step everyone thinks ML is about — but as you've seen, it's step 7 of 10.

Training means:

Feed data into the algorithm — the model sees examples and adjusts its internal parameters (weights) to minimize error
Split data into train/validation/test sets — typically 70/15/15 or 80/10/10. Never evaluate on data the model trained on.
Choose a loss function — the mathematical definition of "what is wrong." Cross-entropy for classification, MSE for regression, etc.
Set hyperparameters — learning rate, batch size, epochs, regularization strength. These are not learned by the model; you set them.
Iterate — training is rarely one-shot. You train, look at results, adjust, retrain.

Key concept: train/test split. If you evaluate your model on the same data it trained on, you get misleadingly high scores. It's like grading a student using the exact exam questions they practiced on.

8. Model Evaluation and Tuning — "How is your model doing?"

Training is done. Now: is the model actually good?

Evaluation Metrics

Different problems need different metrics:

Accuracy — % of correct predictions. Misleading with imbalanced data (99% accuracy on fraud detection means nothing if you just predict "not fraud" every time).
Precision — Of all things the model flagged as positive, how many were actually positive?
Recall — Of all actual positives, how many did the model catch?
F1 Score — Harmonic mean of precision and recall. Good when you need to balance both.
AUC-ROC — Area under the curve. Measures how well the model separates classes across all thresholds.
MSE / RMSE / MAE — For regression: how far off are predictions from actual values?

Hyperparameter Tuning

If results aren't good enough, adjust hyperparameters:

Grid Search — try every combination of predefined values
Random Search — randomly sample combinations (often faster than grid search)
Bayesian Optimization — smart search that learns from previous trials

Dealing with Problems

Overfitting (training score high, test score low) → more data, simpler model, regularization, dropout
Underfitting (both scores low) → more complex model, more features, longer training
Class imbalance → oversampling (SMOTE), undersampling, class weights, different metrics

This step often sends you back to steps 3, 4, 5, or 6. That's the iterative nature of ML.

9. Model Deployment — "Integrate model to the real world"

Your model works in a Jupyter notebook. Now it needs to work in production — handling real users, real data, and real scale.

Deployment means:

Packaging the model — save weights, serialize with ONNX, TorchScript, or pickle
Creating an API — wrap the model in a REST API (Flask, FastAPI) or gRPC endpoint
Infrastructure — where does it run? AWS SageMaker, Google Vertex AI, Azure ML, self-hosted Kubernetes, or edge devices?
Scaling — handle 10 requests/second? 10,000? Auto-scaling, load balancing, caching
CI/CD for ML — automated testing, model versioning, rollback capabilities

Common deployment patterns:

Real-time inference — API call, response in milliseconds (fraud detection, chatbot)
Batch inference — process large datasets periodically (weekly churn predictions, nightly recommendations)
Edge deployment — model runs on device (mobile app, IoT sensor, self-driving car)

Deployment is NOT the finish line. It's where the real work begins.

10. Monitoring and Maintenance — "Keep model healthy"

A deployed model is a living system. It degrades over time because the real world changes.

What to Monitor

Model performance — are accuracy/precision/recall staying stable?
Data drift — is incoming data different from training data? (seasonal changes, new user demographics, market shifts)
Concept drift — has the relationship between features and target changed? (what predicted churn in 2023 might not in 2026)
Latency — is inference speed within requirements?
Resource usage — CPU, memory, cost

When to Retrain

Performance drops below a threshold
Data distribution shifts significantly
Business requirements change
New data categories appear that the model has never seen

Best Practices

Set up automated alerts for performance degradation
Keep a champion/challenger system: new model version runs alongside the old one; switch only when the new one proves better
Log everything: predictions, input data, confidence scores. You'll need this for debugging.
Version your models like you version code. Know exactly which model version produced which prediction.

Monitoring is vital. A model that was 95% accurate at launch can silently drop to 70% if nobody's watching.

Summary — The Key Takeaways

ML is an iterative and structured process. It's a cycle, not a line. You will loop back to earlier steps repeatedly — that's normal, not failure.

Data quality and feature engineering are critical. Steps 3 and 5 have more impact on final model performance than the choice of algorithm at step 6. Good features beat fancy models.

Evaluation and tuning improve model performance. Don't ship the first model that trains. Rigorously evaluate, tune hyperparameters, and test on data the model has never seen.

Deployment isn't the end; monitoring is vital. The real world changes. Your model will degrade. Monitor, retrain, and iterate continuously.

The lifecycle is a loop. The best ML teams are the ones that spin through it fastest — not the ones with the fanciest models.

*If this helped you see the full picture of ML beyond "just training," drop a reaction.

Learn Kubernetes the Manga Way

Aoi Takahashi — Sat, 25 Apr 2026 09:08:28 +0000

Do you want to learn Kubernetes in a fun way?

I wrote a manga (comic book) about Kubernetes back in 2019 — in Japanese.

Recently, I gave a talk at KubeCon + CloudNativeCon Europe 2026, and it inspired me to finally translate it into English.

I shared printed copies with friends at the conference, and the response was amazing!

So here we are — I'm making the English PDF available online. Stay tuned for the download link!

If you want to read rest of the comic, please access to the page!

Free to download!

Learn Kubernetes the Manga Way - Payhip

A beginner-friendly Kubernetes manga featuring Otofu-kun, a cute cat guide.First published in 2019. The core concepts still hold up, though some specifics may reflect older versions.Free to download — enjoy!36p PDF

payhip.com

AI actually made my tutorial addiction worse

Samaresh Das — Sat, 25 Apr 2026 09:00:52 +0000

AI didn't just fail to cure my tutorial addiction; it became its most enthusiastic enabler. What I thought would be a shortcut to understanding quickly morphed into an endless rabbit hole of half-baked knowledge and a constant craving for just one more explanation.

Like many developers, I often find myself falling into the tutorial trap: watching, reading, and copying code without fully internalizing the underlying concepts. When AI assistants like ChatGPT exploded onto the scene, my immediate thought was, "Finally, an instant answer machine!" I envisioned swift solutions, bypassing the need to sift through countless articles. Boy, was I wrong.

My initial approach was to ask AI how to build specific features or integrate complex libraries. It would dutifully spit out boilerplate code, often quite impressive for a first pass. I'd copy it, get it "working" (sometimes), and feel a fleeting sense of accomplishment.

Here’s the catch: the moment something went slightly off, or I needed to customize beyond the generated snippet, I was utterly lost. The AI had given me the what, but never the why. It was like being given a perfectly assembled IKEA cabinet without the instructions or the tools to fix it if a screw came loose. My "understanding" was superficial at best.

Take, for instance, trying to implement a complex authentication flow with a lesser-known framework. I'd prompt the AI:

// Prompt: "How to set up JWT auth with Node.js and Passport.js, storing tokens in localStorage?"

// AI might give you something like this for a route:
app.post('/login', passport.authenticate('local', { session: false }), (req, res) => {
  const token = jwt.sign(req.user.toJSON(), 'your_jwt_secret');
  res.json({ user: req.user, token });
});

// But then you'd be left wondering:
// - How does `passport.authenticate` actually work under the hood?
// - What are 'local' strategies? How do I write one?
// - How do I secure the 'your_jwt_secret' in a real app?
// - What about refresh tokens, token expiration, blacklisting?
// - Is storing in localStorage even secure? (Spoiler: usually not for JWTs!)

This led to a new cycle: I'd take the AI's code, hit a roadblock, and then have to find another tutorial to explain the specific part I didn't grasp. It wasn't replacing tutorials; it was multiplying the need for them, albeit for more granular issues. My "instant answer machine" became a tutorial-request-generator.

The core issue is that AI excels at synthesis and retrieval, not genuine understanding. It stitches together patterns from its training data. For a developer, truly learning means building a mental model of how systems interact, understanding trade-offs, and debugging complex errors – skills that copying AI-generated code snippets simply doesn't develop. It creates an illusion of productivity while stifling deeper learning.

The clear takeaway here is that foundational understanding is irreplaceable. AI is a fantastic tool for brainstorming, boilerplate generation, or even debugging hints, but it's not a substitute for rolling up your sleeves and genuinely learning the concepts.

As someone who builds websites and freelances, I've learned that truly delivering value means understanding the foundations, not just patching things together with AI-generated snippets. If you're looking for a dev who digs deep, check out my work at https://hire-sam.vercel.app/

Share this with your dev friends if you've felt the same way!

AI #WebDev #DeveloperLife #TutorialHell #LearningToCode

Plausible vs Fathom Analytics: Which Fits in 2026?

Juan Diego Isaza A. — Sat, 25 Apr 2026 09:00:48 +0000

If you’re searching for plausible vs fathom analytics, you’re probably trying to ship a privacy-friendly dashboard without turning your site into a surveillance project—or paying enterprise prices for basic pageview stats. Both tools promise “simple analytics,” but the differences show up fast once you care about accuracy under ad blockers, event tracking, and how much you want to own your data.

What Plausible and Fathom actually optimize for

Both Plausible and Fathom sit in the “lightweight, privacy-first” corner of the ANALYTICS_TOOLS world: minimal JavaScript, no cookies by default, and dashboards that don’t require a PhD.

Plausible tends to optimize for:

Open-source transparency and self-hosting as a first-class option
A slightly more “builder-friendly” product surface (goals, events, filtering)
Teams that want credible marketing attribution without invasive tracking

Fathom tends to optimize for:

Extreme simplicity (fewer knobs, fewer ways to misconfigure)
Strong “set it and forget it” ergonomics
A product philosophy that’s aggressively anti-creepiness

Opinionated take: if you’re the type who reads changelogs and cares about data plumbing, Plausible usually feels more extensible. If you want analytics to disappear into the background, Fathom is hard to beat.

Privacy, compliance, and the “no cookies” reality

The main reason teams switch to these tools is to reduce compliance and user-trust friction.

Cookie-less defaults: Both aim to work without cookies by default, which can simplify consent prompts in many setups. (Still: always validate against your legal requirements and region.)
Data minimization: Both tools avoid collecting the kinds of identifiers that make regulators—and users—uneasy.
Self-hosting vs SaaS: Plausible’s open-source positioning makes self-hosting a common path. Fathom is more commonly used as a hosted service.

The trade-off is obvious: less tracking means less ability to stitch sessions across devices or do user-level retention analysis. If your product needs that, you’re in Amplitude/mixpanel territory—and you’re also signing up for a very different privacy posture.

Tracking depth: pageviews vs events (and when you outgrow them)

For many sites, pageviews + referrers + top pages is enough. But product teams quickly ask:

“Which CTA drives signups?”
“How many people hit the error state?”
“Did the new onboarding reduce drop-off?”

Both tools support event/goal tracking, but they intentionally stop short of “full product analytics.” That’s the point.

A practical way to think about it:

If you mostly run content + marketing sites, you’ll likely be happy with either.
If you run a SaaS app and need funnels, cohorts, or user paths, you might eventually graduate to PostHog (still privacy-conscious, but heavier) or go full enterprise with Amplitude/mixpanel.
If you need qualitative context (rage clicks, scroll depth, session replay), that’s Hotjar or FullStory—and again, different trade-offs.

Opinionated take: don’t force Plausible/Fathom to become “poor man’s Amplitude.” You’ll spend time reinventing what specialized tools already solved.

Implementation: a minimal event you can ship today

Below is a simple pattern you can use to track a signup CTA click with either tool’s “custom event” approach. The exact function name differs depending on what script you installed, but the idea is the same: send an event without attaching personal data.

<button id="signup">Start free trial</button>
<script>
  document.getElementById('signup').addEventListener('click', () => {
    // Example custom event call (adjust to your provider’s API)
    window.plausible?.('SignupCTA_Click', { props: { location: 'hero' } });
    // or: window.fathom?.trackGoal('SIGNUP', 0);
  });
</script>

Rules of thumb that keep your analytics useful:

Name events like products, not like engineers. SignupCTA_Click beats btn_1_click.
Avoid personal data in props. Don’t pass emails, user IDs, or anything you can regret later.
Track decisions, not everything. If you can’t answer “what will we change if this moves,” don’t track it.

Choosing between Plausible and Fathom (and what to pair them with)

Here’s the decision I’d make in real projects:

Choose Plausible if you want:
- Open-source comfort and an easier path to self-host
- More flexibility in slicing data and configuring goals
- A setup that can grow with a small engineering team
Choose Fathom if you want:
- The simplest dashboard your stakeholders will actually open
- Minimal configuration and fewer “analytics bikeshed” debates
- A strong bias toward doing less—and doing it well

Neither is “better,” but they are different in the kinds of teams they reward.

Soft recommendation to close: if you start with Plausible or Fathom for clean, privacy-first measurement and later realize you need deeper product analytics, you can layer in something like PostHog for funnels/cohorts—while keeping your public-site analytics lightweight and respectful.

N+1 Query Detection and Prevention in Laravel Production Apps

Deploynix — Sat, 25 Apr 2026 09:00:06 +0000

The N+1 query problem is the most common performance issue in Laravel applications, and it's the easiest to introduce accidentally. A developer adds $post->author->name in a Blade template, and suddenly a page that loaded 10 posts now executes 11 database queries instead of 2. Scale that to 100 posts and you have 101 queries. Scale to a thousand and your database server is on fire.

What makes N+1 problems particularly insidious is that they're invisible during development. Your local database with 20 records responds instantly whether you execute 1 query or 100. The problem only becomes apparent in production where tables have millions of rows, query latency is higher, and database connections are shared across hundreds of concurrent requests.

This post covers everything you need to detect N+1 queries, prevent them from being introduced, and monitor for them in production Laravel applications.

Understanding the N+1 Problem

An N+1 query occurs when you load a collection of N models and then access a relationship on each model individually, triggering a separate query for each one.

Here's the classic example:

// Controller
$posts = Post::all(); // 1 query: SELECT * FROM posts

// Blade template
@foreach($posts as $post)
    {{ $post->author->name }} // N queries: SELECT * FROM users WHERE id = ?
@endforeach

The first query loads all posts. Then, for each post, Laravel lazy-loads the author relationship with a separate query. With 50 posts, that's 51 queries total.

The fix is eager loading — tell Laravel to load all authors in a single query upfront:

$posts = Post::with('author')->get(); // 2 queries total
// Query 1: SELECT * FROM posts
// Query 2: SELECT * FROM users WHERE id IN (1, 2, 3, ...)

Two queries instead of 51. The concept is simple. The challenge is applying it consistently across a growing codebase where relationships are accessed in controllers, views, components, API resources, and queued jobs.

Prevention: Laravel's preventLazyLoading

Laravel provides a built-in mechanism to catch N+1 problems during development. When enabled, any lazy-loaded relationship throws an exception:

// In a service provider's boot method
use Illuminate\Database\Eloquent\Model;

Model::preventLazyLoading(! app()->isProduction());

Now the $post->author->name call without eager loading throws a LazyLoadingViolationException instead of silently executing an extra query. This forces developers to explicitly eager-load every relationship they access.

Handling Violations in Production

You don't want exceptions crashing pages in production. Instead, log violations so you can fix them without disrupting users:

Model::preventLazyLoading();

Model::handleLazyLoadingViolationUsing(function ($model, $relation) {
    logger()->warning("N+1 detected: lazy loading [{$relation}] on [" . get_class($model) . "]");
});

This approach gives you the best of both worlds: hard failures in development that force immediate fixes, and silent logging in production that reveals problems you missed.

When preventLazyLoading Gets in the Way

There are legitimate cases where lazy loading is acceptable:

Artisan commands that process a single model and access one relationship
Queue jobs that operate on a known small dataset
Tinker sessions during debugging

You can selectively allow lazy loading on specific models:

// On a specific model
class Setting extends Model
{
    // Settings table is tiny; lazy loading is fine
    public $preventLazyLoading = false;
}

Or disable it temporarily in specific contexts:

Model::withoutPreventing(function () {
    // Lazy loading is allowed in this closure
    $user->profile->bio;
});

Eager Loading Patterns

Basic Eager Loading

The with() method accepts a single relationship or an array:

$posts = Post::with(['author', 'tags', 'comments'])->get();

Nested Eager Loading

Load relationships of relationships using dot notation:

$posts = Post::with(['comments.author', 'author.profile'])->get();
// Loads: posts, comments, comment authors, post authors, author profiles

Constrained Eager Loading

Filter or limit the eager-loaded relationship:

$posts = Post::with(['comments' => function ($query) {
    $query->where('approved', true)
          ->latest()
          ->limit(5);
}])->get();

Laravel 12 supports limit() on eager loads natively — no external packages needed.

Conditional Eager Loading

Load relationships only when certain conditions are met:

$posts = Post::query()
    ->when($includeComments, fn ($query) => $query->with('comments'))
    ->when($includeAuthor, fn ($query) => $query->with('author'))
    ->get();

Eager Loading on Existing Collections

If you already have a collection and need to load a relationship after the fact:

$posts = Post::all();

// Later, you realize you need authors
$posts->load('author');

// Or load only if not already loaded
$posts->loadMissing('author');

loadMissing() is particularly useful in deep call stacks where you're not sure if a relationship was already eager-loaded upstream.

Default Eager Loading on Models

If a relationship is always needed when a model is loaded, define it as a default:

class Post extends Model
{
    protected $with = ['author'];
}

Use this sparingly. It means every Post::all(), Post::find(), and Post::where(...) call loads the author relationship, even when it's not needed. This can turn a simple count query into a heavy join operation.

A better approach is to define scopes for common access patterns:

class Post extends Model
{
    public function scopeWithFeedRelations(Builder $query): Builder
    {
        return $query->with(['author', 'tags', 'comments' => function ($query) {
            $query->latest()->limit(3);
        }]);
    }
}

// Usage
$posts = Post::withFeedRelations()->latest()->paginate(20);

Detecting N+1 in Existing Code

Query Counting in Tests

Write tests that enforce query budgets:

it('loads the post index without N+1 queries', function () {
    $author = User::factory()->create();
    Post::factory()->count(20)->for($author, 'author')->create();

    $queryCount = 0;
    DB::listen(function () use (&$queryCount) {
        $queryCount++;
    });

    $this->actingAs($author)
        ->get('/posts')
        ->assertOk();

    expect($queryCount)->toBeLessThan(10);
});

If loading 20 posts requires more than 10 queries, something is lazy loading when it shouldn't be. This test catches regressions when someone adds a new relationship access in the view without updating the controller's eager loading.

Using Laravel Debugbar

Debugbar shows duplicate queries in its queries tab. If you see SELECT * FROM users WHERE id = ? repeated 20 times with different IDs, that's an N+1 problem on the author relationship.

Debugbar also shows the file and line where each query was triggered. This makes it trivial to trace the lazy load back to the specific Blade template line or component that caused it.

Using Telescope's Query Tab

Telescope groups queries by request. Look for requests with high query counts — anything above 20-30 queries per request warrants investigation. Sort by query count to find the worst offenders first.

Common N+1 Traps

Trap 1: Blade Components That Access Relationships

// app/View/Components/PostCard.php
public function render(): View
{
    return view('components.post-card');
}

// components/post-card.blade.php



## {{ $post->title }}

    by {{ $post->author->name }}  {{-- N+1 if author not eager-loaded --}}

The component looks harmless. The N+1 is hidden because the relationship access happens in the view, far from the controller that loaded the posts.

Fix: Ensure the parent view or controller eager-loads the relationship:

$posts = Post::with('author')->paginate(20);

Trap 2: API Resources with Nested Relationships

class PostResource extends JsonResource
{
    public function toArray(Request $request): array
    {
        return [
            'id' => $this->id,
            'title' => $this->title,
            'author' => new UserResource($this->author), // N+1
            'comments_count' => $this->comments->count(), // N+1 AND loads all comments
        ];
    }
}

Fix: Eager-load in the controller and use withCount for counts:

$posts = Post::with('author')->withCount('comments')->paginate(20);

// In the resource:
'comments_count' => $this->comments_count, // Uses the aggregate, no extra query

Trap 3: Accessors That Touch Relationships

class Order extends Model
{
    protected function totalWithTax(): Attribute
    {
        return Attribute::get(fn () => $this->total * (1 + $this->taxRate->rate));
        // Accessing $this->taxRate triggers a lazy load
    }
}

Accessors are called during serialization, in Blade templates, and in API resources. Each call triggers the relationship query.

Fix: Eager-load taxRate wherever you use this accessor, or cache the relationship:

protected function totalWithTax(): Attribute
{
    return Attribute::get(function () {
        $rate = $this->relationLoaded('taxRate')
            ? $this->taxRate
            : $this->taxRate()->first();

        return $this->total * (1 + $rate->rate);
    });
}

Trap 4: Queued Jobs Processing Collections

class SendWeeklyDigest implements ShouldQueue
{
    public function handle(): void
    {
        User::where('digest_enabled', true)->chunk(100, function ($users) {
            foreach ($users as $user) {
                // Each of these triggers a query
                $recentPosts = $user->subscriptions->flatMap->posts;
                Mail::to($user)->send(new WeeklyDigest($recentPosts));
            }
        });
    }
}

Fix:

User::where('digest_enabled', true)
    ->with(['subscriptions.posts' => function ($query) {
        $query->where('created_at', '>', now()->subWeek());
    }])
    ->chunk(100, function ($users) {
        foreach ($users as $user) {
            $recentPosts = $user->subscriptions->flatMap->posts;
            Mail::to($user)->send(new WeeklyDigest($recentPosts));
        }
    });

Monitoring in Production

Even with preventLazyLoading and test coverage, N+1 problems can slip through. Monitor query counts in production using multiple layers.

Laravel Pulse

Pulse tracks slow queries and high-frequency queries in production. A query that appears hundreds of times per minute with slight parameter variations is likely an N+1 problem. Check Pulse's slow queries dashboard regularly.

Deploynix Server Monitoring

On Deploynix, monitor your database server's CPU and connection count. A sudden increase in active connections often correlates with N+1 queries — each lazy-loaded query opens and closes a connection (or holds one from the pool). If your connection count spikes during peak traffic, investigate the most active endpoints for N+1 issues.

Deploynix's real-time monitoring dashboard shows MySQL process CPU usage alongside your application server's metrics. When the database CPU spikes but the application CPU stays flat, the cause is almost always excessive queries — and N+1 is the most likely culprit.

Custom Middleware for Query Counting

Add middleware that logs requests exceeding a query threshold:

class QueryCountMiddleware
{
    public function handle(Request $request, Closure $next): Response
    {
        $queryCount = 0;
        DB::listen(function () use (&$queryCount) {
            $queryCount++;
        });

        $response = $next($request);

        if ($queryCount > 30) {
            logger()->warning("High query count: {$queryCount} queries", [
                'url' => $request->fullUrl(),
                'method' => $request->method(),
            ]);
        }

        return $response;
    }
}

Conclusion

N+1 queries are the single most impactful performance problem in Laravel applications, and they're entirely preventable. Enable preventLazyLoading to catch them during development. Write query-count tests to prevent regressions. Use with(), loadMissing(), and withCount() consistently.

In production, monitor query patterns through Pulse and watch for database CPU spikes on your Deploynix dashboard. The combination of prevention at the code level and detection at the infrastructure level means N+1 problems get caught before they become outages.

The goal isn't zero lazy loading — it's intentional lazy loading. Every relationship access should be a deliberate choice, not an accidental database round trip hiding in a Blade component three levels deep.

Advanced PostgreSQL: JSONB, partial indexes and partitioning

Odilon HUGONNOT — Sat, 25 Apr 2026 09:00:03 +0000

A trades table with 50 million rows. The query "which active trades for this user this month" takes 4 seconds. After applying the four techniques in this article: 12 ms.

This isn't magic — it's a solid understanding of what PostgreSQL actually does under the hood. I already covered slow query diagnosis with EXPLAIN ANALYZE basics. Here we go further: the four levers that make a real difference on high-volume production tables.

JSONB: storing flexible schemas without sacrificing performance

Every crypto exchange exposes a different API. Binance returns a clientOrderId, Kraken a userref, Coinbase a client_order_id nested inside a sub-object. If you create a dedicated column for each field from each exchange, your table ends up with 40 columns where 35 are NULL for every single row.

JSONB solves this problem — provided you understand when to use it.

`json` vs `jsonb` — always use `jsonb`

json stores the raw text as-is. jsonb decomposes it into a binary representation at insert time. The consequences:

jsonb is indexable (GIN, expression indexes)
jsonb supports containment operators (@>, ?)
jsonb is slightly slower to write, but significantly faster to read

Simple rule: always use jsonb.

Essential operators

-- Access a key (returns jsonb)
SELECT exchange_meta -> 'exchange' FROM trades;

-- Access a key (returns text)
SELECT exchange_meta ->> 'exchange' FROM trades;

-- Containment: does the JSON contain this sub-object?
SELECT * FROM trades WHERE exchange_meta @> '{"exchange": "binance"}';

-- Key existence
SELECT * FROM trades WHERE exchange_meta ? 'client_order_id';

Adding the column and GIN index

-- Store exchange-specific metadata without extra columns
ALTER TABLE trades ADD COLUMN exchange_meta JSONB;

-- GIN index for containment queries (@> operator)
CREATE INDEX idx_trades_exchange_meta ON trades USING GIN (exchange_meta);

-- Query: all trades with a specific exchange order ID
SELECT * FROM trades WHERE exchange_meta @> '{"order_id": "12345"}';

-- Query: all Binance trades with status filled
SELECT * FROM trades WHERE exchange_meta @> '{"exchange": "binance", "status": "filled"}';

The GIN index on exchange_meta automatically covers all keys in your documents. No need to anticipate upfront which keys you'll query against.

When not to use JSONB

JSONB is not an excuse to skip data modeling. If you regularly filter on a field — user_id, status, created_at — that field must be a real column with a proper B-tree index. Putting those fields inside JSON forfeits all planner optimizations.

Rule: JSONB for variable data that is rarely filtered directly. Typed columns for everything that is frequently filtered, sorted, or joined.

Partial indexes: only index what matters

This is the most underused PostgreSQL feature. A partial index only covers rows that satisfy a WHERE clause. Its size can be 100x smaller than a full index — and it fits entirely in RAM.

The problem with full indexes

-- Full index on status: indexes ALL 50 MILLION rows,
-- including the 49.9 million closed trades nobody queries
CREATE INDEX idx_trades_status ON trades(status);

-- Approximate size
SELECT pg_size_pretty(pg_relation_size('idx_trades_status'));
-- → 1 GB

The solution: partial index on active rows only

-- Partial index: only the ~100,000 active trades
CREATE INDEX idx_trades_active ON trades(user_id, created_at)
WHERE status = 'active';

-- Actual size
SELECT pg_size_pretty(pg_relation_size('idx_trades_active'));
-- → 3 MB  ← fits entirely in shared memory

-- This query uses the partial index directly
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
ORDER BY created_at DESC;

PostgreSQL knows that a query with WHERE status = 'active' can use this index — the index predicate is a subset of the query condition. The planner selects the partial index automatically.

Other common use cases

-- Unverified emails (minority of users)
CREATE INDEX idx_users_unverified ON users(created_at)
WHERE verified = false;

-- Pending jobs in a queue
CREATE INDEX idx_jobs_pending ON jobs(priority, created_at)
WHERE status = 'pending';

-- Open orders
CREATE INDEX idx_orders_open ON orders(user_id, amount)
WHERE closed_at IS NULL;

The principle is always the same: identify the subset of rows your hot queries target, and index only that subset.

EXPLAIN ANALYZE: reading between the lines

"Seq Scan bad, Index Scan good" — that's the naive reading. The reality is more nuanced. Here's how to read an execution plan correctly.

The full command

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
ORDER BY created_at DESC
LIMIT 20;

ANALYZE actually executes the query and measures real timings. BUFFERS reports cache and disk activity. Never run this on a DELETE or UPDATE without wrapping it in a transaction.

Decoding the plan

Limit  (cost=0.56..18.42 rows=20 width=112) (actual time=0.124..0.198 rows=20 loops=1)
  ->  Index Only Scan Backward using idx_trades_active on trades
        (cost=0.56..2891.33 rows=3201 width=112)
        (actual time=0.121..0.183 rows=20 loops=1)
      Index Cond: (user_id = 123)
      Heap Fetches: 0
      Buffers: shared hit=5 read=0
Planning Time: 0.312 ms
Execution Time: 0.221 ms

The key nodes:

Index Only Scan — the best case. PostgreSQL reads the index without touching the heap (the physical table). Heap Fetches: 0 confirms zero table access. This is possible because all needed columns are covered by the index.
rows=3201 estimated vs rows=20 actual — a moderate discrepancy, acceptable. A 100x divergence means stale statistics; run ANALYZE trades;.
Buffers: shared hit=5 read=0 — everything served from memory cache, zero disk I/O. If read is high, your working set doesn't fit in RAM.

Pattern to watch for: the expensive Sort node

Sort  (cost=15234.12..15489.23 rows=102044 width=112)
      (actual time=1823.44..2104.12 rows=102044 loops=1)
  Sort Key: created_at DESC
  Sort Method: external merge  Disk: 8432kB

external merge Disk means the sort spilled to disk — work_mem is insufficient, or the index should include the sort column. Fix: include created_at in the index so rows come out pre-sorted.

-- Before: index without sort order
CREATE INDEX idx_trades_user ON trades(user_id) WHERE status = 'active';

-- After: composite index with native sort direction
CREATE INDEX idx_trades_user_date ON trades(user_id, created_at DESC)
WHERE status = 'active';
-- → The Sort node disappears from the plan

Date partitioning: the solution for massive tables

Beyond 10 million rows on an append-only table (trades, logs, events), date-range partitioning becomes necessary. The idea: physically split the table into sub-tables by time period. A query for "this month" only scans one partition instead of the whole thing.

Creating the partitioned table

-- Parent table — defines the structure and partition key
CREATE TABLE trades (
    id          BIGSERIAL,
    user_id     BIGINT       NOT NULL,
    pair        TEXT         NOT NULL,
    amount      NUMERIC(20, 8) NOT NULL,
    status      TEXT         NOT NULL DEFAULT 'active',
    exchange_meta JSONB,
    created_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (created_at);

-- Monthly partitions
CREATE TABLE trades_2026_01 PARTITION OF trades
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE trades_2026_02 PARTITION OF trades
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE trades_2026_03 PARTITION OF trades
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

Partition pruning in action

EXPLAIN SELECT * FROM trades
WHERE created_at >= '2026-03-01'
  AND created_at < '2026-04-01'
  AND user_id = 123;

Append  (cost=0.29..45.23 rows=18 width=156)
  Partitions: trades_2026_03
  ->  Index Scan using trades_2026_03_user_id_idx on trades_2026_03
        (cost=0.29..45.23 rows=18 width=156)
      Index Cond: (user_id = 123)

Partitions: trades_2026_03 — PostgreSQL eliminated all other partitions at planning time. It physically accesses only March 2026.

Operational benefits

Instant archiving: DROP TABLE trades_2024_01 removes a full month of data without a massive DELETE or table lock.
Targeted VACUUM: autovacuum runs partition by partition. Frozen old partitions are never reprocessed.
Local indexes: indexes created on the parent table are automatically created on each partition — each one smaller, faster to build, and more likely to fit in memory.

-- Creating an index applies to all existing and future partitions
CREATE INDEX ON trades(user_id, created_at DESC) WHERE status = 'active';

-- Archive January 2024 without locking the table
DROP TABLE trades_2024_01;

-- Or detach first, then archive
ALTER TABLE trades DETACH PARTITION trades_2024_01;

Combining all four

Here is the final state of the table after applying all four techniques:

-- 1. Table partitioned by month
CREATE TABLE trades (
    id            BIGSERIAL,
    user_id       BIGINT         NOT NULL,
    pair          TEXT           NOT NULL,
    amount        NUMERIC(20, 8) NOT NULL,
    status        TEXT           NOT NULL DEFAULT 'active',
    exchange_meta JSONB,                          -- 2. JSONB for variable metadata
    created_at    TIMESTAMPTZ    NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (created_at);

-- Monthly partitions (automate with pg_partman in production)
CREATE TABLE trades_2026_03 PARTITION OF trades
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

-- 3. Partial index: only active trades, composite with sort direction
CREATE INDEX idx_trades_active ON trades(user_id, created_at DESC)
WHERE status = 'active';

-- 4. GIN index for containment queries on exchange_meta
CREATE INDEX idx_trades_exchange_meta ON trades USING GIN (exchange_meta);

The query that used to take 4 seconds:

-- Before: Seq Scan over 50M rows, 4 seconds
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
  AND created_at >= date_trunc('month', NOW())
ORDER BY created_at DESC
LIMIT 20;

-- After: partition pruning + composite partial index
-- Execution Time: 12 ms

Index Only Scan Backward using idx_trades_active on trades_2026_03
  Index Cond: (user_id = 123)
  Filter: (created_at >= date_trunc('month', now()))
  Heap Fetches: 0
  Buffers: shared hit=4 read=0

The gain comes from cumulative effect: partitioning narrows the scope to a single monthly partition, the partial index only covers active trades within that partition, and the index column order eliminates the sort. PostgreSQL touches only the 4 memory pages it strictly needs.

Conclusion

The logical optimization order for a high-volume table:

Partial indexes first: immediate gain, zero schema change, spectacular impact on hot queries targeting a data subset.
JSONB for variable data: when external API responses have unpredictable fields, JSONB with a GIN index avoids a proliferation of NULL columns. It does not replace typed columns for frequently filtered fields.
Partitioning when the table is truly large: from 10M rows on append-only data. The upfront investment is higher (migration, partition automation, pg_partman), but the long-term operational benefits are substantial.

And at every step, EXPLAIN (ANALYZE, BUFFERS) to verify the planner is doing what you expect. Row estimates and the hit/read ratio don't lie.

Navigating the Magento Migration Maze: Developer Shortages, AI, and Smart Agency Operations

EShopSet — Sat, 25 Apr 2026 09:00:03 +0000

Attention, agency owners, project managers, and developers – let's discuss a topic that has generated considerable buzz within the ecommerce community: Magento migrations. Specifically, we're talking about the widely perceived shortage of skilled Magento developers and the concurrent surge in complex replatforming projects. We recently encountered a vibrant online discussion that perfectly illustrates this current landscape, offering significant insights into how you should manage your ecommerce delivery workflow.

The individual who initiated the discussion began by emphasizing the persistent challenge of finding proficient Magento developers, alongside an apparent increase in migration projects. Their proposed solution involved hiring junior developers and strategically utilizing AI to construct Magento projects, surprisingly claiming successful outcomes. This indeed represents a bold approach, and as might be expected, it ignited a considerable debate among the participants.

In-content image: Ecommerce agency team managing a complex migration project.## The AI & Junior Dev Debate: A Double-Edged Sword?

The concept of deploying junior developers, particularly when augmented by artificial intelligence, to undertake intricate Magento projects immediately provoked skepticism. One community member articulated a prevalent concern: the potential for accumulating substantial technical debt. Magento, as another respondent astutely observed, is not a platform renowned for being "vibe code friendly." Instead, it necessitates a profound understanding, meticulous structural integrity, and adherence to robust development practices.

Several seasoned developers contributed to the conversation, reinforcing this perspective. One expert specifically cautioned that AI, despite its immense capabilities, can introduce complications even for highly experienced FANG developers, urging significant prudence. The prevailing sentiment indicated that while AI-assisted coding can serve as a valuable instrument for enhancing productivity and guiding emerging talent, relying on AI for "vibe coding" or generating entire complex Magento projects without experienced oversight is almost certainly a precursor to failure. The core principle lies in mindset and priorities: AI should function as a co-pilot, not the lead engineer, especially when managing client funds and critical business logic.

The long-term implications for maintenance also emerged as a significant point of discussion. While initiating a project with junior talent and AI might seem straightforward, the true challenge arises years later when technical debt inevitably surfaces. As one community member accurately articulated, "It's easy to start building, but hard to maintain with juniors only." This statement profoundly underscores the critical necessity for robust ecommerce replatforming project management that consistently prioritizes enduring stability over mere short-term velocity.

Beyond Code: Safeguarding SEO and Scaling Operations

The scarcity of developers constitutes only one facet of the challenge. A crucial insight shared by an experienced community member highlighted an even greater risk associated with employing junior developers for migrations: the detrimental SEO consequences stemming from a poorly executed URL mapping strategy. Manual redirects implemented within .htaccess files, though commonly used, frequently lead to latency issues and broken redirect chains as a website expands. This concern transcends mere code quality; it fundamentally involves protecting the client's most valuable asset – their organic search traffic.

The definitive solution involves scripting redirect mapping as an integral component of the deployment pipeline. Automating the precise mapping between legacy Magento URLs and the new site structure represents the only viable method to maintain low 404 rates and prevent core web vitals from plummeting on launch day. This proactive methodology, while admittedly a "grind," ultimately conserves immense time and averts costly SEO recovery efforts. It serves as a prime illustration of a non-negotiable item that must be included on any comprehensive delivery checklists designed for migration projects.

This critical insight extends far beyond simple redirects. Every single aspect of a migration project, ranging from comprehensive data transfer to intricate third-party integrations, mandates a highly structured and systematic approach. An agency operations platform such as EShopSet becomes absolutely indispensable in this context, providing the essential framework for these crucial delivery checklists. This ensures that no critical step is overlooked and that best practices are consistently enforced across all projects, irrespective of the individual developer's specific experience level.

The Conflicting Reality: Developer Availability

Intriguingly, while some agencies report a severe deficit and escalating costs for senior Magento developers, others participating in the discussion expressed difficulties in securing Magento projects themselves. This notable discrepancy suggests a potential misalignment between the types of projects currently available and the specialized skill sets possessed by developers, or perhaps indicates geographical variations in demand. Regardless, agencies must adeptly navigate this intricate landscape by continuously optimizing their internal operational processes and effectively leveraging their existing talent pool.

Senior developers, while representing a significant investment, contribute invaluable experience, particularly in architecting complex solutions and proactively mitigating technical debt. Their specialized expertise is absolutely crucial for overseeing junior teams and guaranteeing that AI-generated code adheres to stringent quality standards. EShopSet comprehends this dynamic, offering specialized tools that empower senior leads to efficiently review, guide, and manage multiple projects simultaneously, thereby maximizing their overall impact across the entire agency.

EShopSet: Your Operations Command Center for Replatforming

Successfully navigating the inherent complexities of Magento migrations, addressing developer shortages, and managing technical debt demands more than just proficient developers; it necessitates a robust and well-defined operational framework. This is precisely where EShopSet excels, serving as your dedicated agency operations platform.

At EShopSet, we firmly believe that successful ecommerce delivery is fundamentally built upon clearly defined processes, transparent communication, and efficient allocation of resources. For agencies tasked with managing intricate projects such as Magento replatforming, our comprehensive platform meticulously provides:

- **Standardized [Delivery Checklists](#):** Guarantee that every crucial stage, spanning from meticulous SEO redirect mapping to comprehensive data migration and seamless third-party integration, is thoroughly planned and flawlessly executed.

- **Streamlined [Ecommerce Replatforming Project Management](#):** Unify all project plans, individual tasks, and team communications, establishing a singular, reliable source of truth and fostering unambiguous accountability.

- **Technical Debt Mitigation:** Through the enforcement of industry best practices and providing essential oversight, EShopSet actively assists agencies in minimizing the build-up of technical debt, even when collaborating with teams of varied experience levels.

- **Resource Optimization:** Strategically deploy your invaluable senior talent into critical oversight positions, effectively utilizing junior developers and AI for specialized tasks while rigorously upholding quality control standards.

Furthermore, these operational efficiencies are absolutely vital for an agency's overarching Revenue Operations (RevOps) strategy. A project delivery that is expertly managed directly influences client satisfaction, enhances client retention, and ultimately, significantly impacts your agency's financial performance. While EShopSet primarily concentrates on optimizing the delivery workflow, it seamlessly complements your existing CRM systems, such as HubSpot. It achieves this by ensuring that the operational execution aligns perfectly with client expectations and your established sales pipeline. HubSpot's robust CRM, Sales Hub, and Commerce capabilities are indispensable for effectively managing client relationships and driving business development; EShopSet, in turn, guarantees that the actual project delivery phase is equally robust and transparent, feeding valuable data directly back into your comprehensive RevOps strategy. This seamless integration of client management (CRM) and meticulous project delivery (EShopSet) is the cornerstone for successfully scaling any ecommerce agency.

Actionable Takeaways for Your Agency

As the dynamic ecommerce landscape continues its constant evolution, agencies must proactively adapt to emerging challenges. Here’s a strategic guide on how you can fortify your operational capabilities:

- **Invest in Robust Project Management:** Establish comprehensive [delivery checklists](#) and detailed project plans for every migration and replatforming initiative your agency undertakes.

- **Leverage AI Strategically:** Employ AI as a supportive assistant for junior developers, rather than allowing it to replace essential senior oversight. Prioritize AI-*assisted* coding for enhanced efficiency, avoiding "vibe coding" for mission-critical systems.

- **Prioritize SEO and Technical Architecture:** From the project's inception, construct a scalable redirect architecture and guarantee that technical SEO best practices are fully integrated into your [ecommerce replatforming project management](#) framework.

- **Utilize an [Agency Operations Platform](#):** Implement a dedicated platform such as EShopSet to standardize operational workflows, uphold quality standards, and offer the necessary oversight for intricate projects, effectively bridging the divide between client relationship management (often managed in a CRM like HubSpot) and precise project execution.

- **Foster Continuous Learning:** Provide robust support to junior developers through mentorship programs and well-defined structured learning paths, enabling them to expand their expertise and thereby reducing long-term technical debt risks.

The challenges inherent in Magento migrations and the prevailing developer shortages are undeniably real, yet they are entirely surmountable with the implementation of the correct strategy and appropriate tools. By embracing structured processes, intelligent technology, and a specialized platform meticulously designed for ecommerce agencies, you possess the capability to transform these very challenges into significant opportunities for growth, consistently delivering exceptional results for your valued clients.

I built a privacy-first PDF dark mode converter that runs entirely in your browser

1436941541 — Sat, 25 Apr 2026 08:58:01 +0000

TL;DR: I shipped pdfdark.org — a browser-side PDF dark mode converter. Files don't get

uploaded; the entire conversion happens in your browser via PDF.js + a Web Worker. Open source (MIT), free, no signup.

Why I built this

Reading long PDFs at night was killing my eyes. The two paths I had both sucked:

Existing "dark mode PDF" web tools → required uploading the file. For research papers, contracts, medical records

— that felt sketchy. No way to verify what they did with my data.

OS-level "invert colors" → wrecked photos and charts. Faces became X-rays. Graphs became noise.

So I built one with two non-negotiable defaults:

Nothing leaves your browser. PDF.js parses the file, a Web Worker does the dark-mode pass, pdf-lib stitches the

result. Verify it yourself in DevTools → Network.

Images keep their original colors. A saturation classifier detects photos and figures and leaves them untouched

while it darkens text and UI.

How it works

Drop PDF

→ PDF.js renders pages to canvas

→ Web Worker classifies each pixel by saturation

├─ saturated pixels (images, charts) → preserved

└─ low-saturation pixels (text, UI) → themed

→ pdf-lib assembles a new PDF

→ User downloads

The classifier runs on OffscreenCanvas inside a Web Worker, so the UI thread never blocks on large PDFs.

Output is a real PDF (image-based, one JPEG per page), so the dark mode persists when you email it, sync it to iPad, or

open it on a Kindle. Not a viewer toggle.

What it cost to ship

Domain: $7.50 (Cloudflare Registrar, .org)
Hosting: $0 (Vercel free tier)
Email forwarding: $0 (Cloudflare Email Routing)
Error monitoring: $0 (Sentry free tier, errors only with full content masking)
Time: a few weekends

Total cost: $7.50.

What I'm watching

Whether the "no-upload" angle resonates beyond privacy nerds
Edge cases that break the algorithm (weird embedded fonts, scanned PDFs)
Whether to add a vector-preserving mode for text-only PDFs (current output is image-based, so text isn't selectable)

Bulletproofing Your WooCommerce Checkout: A Guide for Agencies on Payment Gateway Resilience

EShopSet — Sat, 25 Apr 2026 08:56:46 +0000

Few things are as alarming for an ecommerce agency owner as the dreaded phrase, "Our client's checkout isn't working!" In the competitive realm of online retail, a disruption in the payment process is more than a mere inconvenience; it directly impacts your client's revenue and can harm your agency's standing. For agencies overseeing numerous client storefronts, establishing robust and resilient payment systems is absolutely crucial.

A recent candid discussion within a community forum perfectly illustrated this common struggle. The original poster, managing a WooCommerce store, recounted a "short issue recently where our main payment method started failing for some customers." This unexpected hiccup resulted in numerous abandoned carts and posed a critical question: How can we construct a more resilient checkout system? Their goal was to integrate backup payment methods, facilitate easy gateway switching, support diverse payment types, leverage payment links for urgent situations, maintain a swift checkout experience, and ultimately, minimize failed payments and cart abandonment.

Black-and-white sketch of a hand holding a phone with a payment link, representing a lifeline for abandoned carts.## The Non-Negotiable: Baseline Payment Gateway Redundancy

The community's consensus was unequivocal: depending solely on a single payment gateway represents an unacceptable risk for any serious ecommerce store. Numerous respondents highlighted Stripe and PayPal as the established industry standard for achieving baseline redundancy. These two platforms can operate concurrently without interference, providing a robust and essential safety net that many agencies wisely integrate for their clientele.

What is frequently overlooked, as one community member pointed out, is that fundamental multi-gateway functionality is already integrated directly into WooCommerce. For the most straightforward configuration, an additional plugin isn't always necessary. Simply navigate to WooCommerce > Settings > Payments, activate both desired gateways, and then reorder them to establish your preferred primary option. This fundamental action is absolutely vital for any agency overseeing a WooCommerce storefront, guaranteeing that should a primary payment processor encounter an outage, a fully functional alternative is instantly accessible.

Reflect on how this directly influences your client's Revenue Operations (RevOps). Each instance of a failed payment, caused by a gateway problem, signifies a direct loss of potential income and a forfeited chance to collect valuable customer data for your HubSpot CRM. By diligently implementing this baseline redundancy, agencies are empowered to substantially mitigate revenue leakage and ensure a steady, uninterrupted stream of transactions, thereby populating sales and marketing funnels with precise and timely data.

Beyond Simple Redundancy: Smart Routing and True Failover

While enabling Stripe and PayPal marks an excellent initial step, a more profound understanding surfaced: merely presenting multiple payment options does not equate to genuine resilience. As one respondent succinctly articulated, "Most stores don't have a backup plan they just have multiple plugins sitting next to each other. The problem is those don't really failover when something breaks they just give users more options upfront which isn't the same as resilience." This distinction is crucial for robust systems.

This underscores a vital distinction for ecommerce developers and agency teams: the fundamental difference between simply offering numerous payment options and actually implementing a genuine failover system. When shoppers encounter an excessive number of payment buttons during checkout, it frequently results in decision paralysis and a rise in cart abandonment, even when all gateways are fully operational. A community member sagely observed that "conversion drops even when both gateways work fine" if the checkout interface becomes overly cluttered.

Authentic resilience, however, incorporates smart routing. This advanced approach means the system intelligently directs transactions towards the most suitable or currently available payment gateway, potentially concealing secondary options unless the primary one experiences a failure. For example, an aggregator such as Mollie, which a community member highlighted, represents a robust solution for stores operating within the EU. It encompasses a wide array of local payment methods (including cards, iDEAL, Bancontact, and Klarna) via a single integration, and expertly manages the internal routing, thereby significantly simplifying the setup process for agencies serving international clients.

Certain advanced solutions, like the one referenced by another contributor, integrate routing and fallback capabilities directly into the core payment flow. This forward-thinking and proactive methodology substantially decreases abandoned carts by automatically and seamlessly switching payment gateways in the background whenever issues emerge. This ensures a significantly smoother customer experience and rigorously safeguards your client's revenue stream from disruptions.

The Emergency Lifeline: Payment Links and Manual Recovery

What course of action should be taken when the entire checkout process on a storefront is compromised, extending beyond a mere single gateway failure? This is precisely when payment links transform into an indispensable emergency resource. A number of community members strongly underscored the efficacy of having pre-generated Stripe Payment Links or PayPal Invoices readily prepared for immediate deployment.

As one contributor wisely advised, possessing a direct URL that can be dispatched to a customer within minutes of an issue arising can result in a remarkably high rate of recovered carts. This particular strategy proves exceptionally effective for agencies overseeing high-value clients, where the successful completion of every single order is paramount. It expertly converts a potentially lost sale into a successfully recovered transaction, thereby showcasing proactive client management and superior problem-solving capabilities.

For agencies, seamlessly integrating this manual recovery procedure into your established operational playbook is absolutely essential. This could entail configuring quick-access templates for your client service teams directly within your agency project hub, empowering them to swiftly generate and dispatch payment links whenever a customer reports a problem. This approach not only salvages potential sales but also significantly strengthens client trust and elevates overall satisfaction.

Proactive Measures: Monitoring and Testing

While redundancy and failover serve as vital reactive solutions, genuine resilience equally necessitates proactive monitoring. As one community member astutely stated, it is imperative to "add basic monitoring so you hear about it before customers do." This encompasses diligently tracking orders that remain in a "pending" status for prolonged durations, meticulously examining webhook delivery logs from Stripe or PayPal, and regularly consulting gateway status pages for any alerts.

For ecommerce developers, establishing automated alerts that seamlessly integrate with your agency's internal communication channels (such as Slack or email) can prove to be a transformative advantage. This critical capability guarantees that your team gains immediate awareness of any potential payment issues, thereby enabling rapid intervention before a significant surge of abandoned carts materializes. Furthermore, consistently testing your backup payment methods is equally crucial to verify their expected functionality precisely when they are most required.

When undertaking an ecommerce replatforming checklist for a client, a comprehensive evaluation of their existing payment infrastructure, which includes rigorously stress-testing backup options and integrating robust monitoring systems, must be considered an absolutely non-negotiable step. This meticulous process guarantees that the newly implemented platform acquires a truly resilient payment system, rather than merely a refreshed storefront.

EShopSet's Role in Managing Payment Complexity for Agencies

Effectively managing resilient payment systems across numerous client accounts often presents significant complexity. It is precisely in this scenario that an operations workspace such as EShopSet proves to be indispensable. Our advanced platform functions as a central agency project hub, empowering you to:

- **Standardize Payment Strategies:** Establish, define, and consistently implement optimal best practices for payment redundancy and failover across the entirety of your client stores.

- **Streamline Incident Response:** Centralize all monitoring alerts and formulate clear, actionable protocols for addressing payment-related issues, thereby ensuring swift and effective resolution.

- **Enhance Client Transparency:** Leverage a dedicated [role based access control client portal](https://www.eshopset.com/role-based-access-control-client-portal) to securely share detailed payment performance reports and concise incident summaries with clients, fostering trust and actively demonstrating proactive management without necessitating full backend access.

- **Integrate with HubSpot:** Guarantee that your robust and well-defined payment strategies seamlessly feed into HubSpot CRM, Sales Hub, and Commerce platforms, delivering a comprehensive and holistic perspective on customer journeys and overall revenue operations.

Developing a truly resilient WooCommerce payment system is no longer merely a luxury; it has become an absolute necessity for ecommerce agencies dedicated to safeguarding both client revenue and their own professional reputation. By conscientiously adopting baseline redundancy, implementing intelligent smart routing, preparing emergency payment links, and engaging in proactive monitoring, agencies possess the capability to transform potential checkout catastrophes into effortlessly seamless transactions. Furthermore, with the aid of powerful tools like EShopSet, the management of this inherent complexity is significantly simplified, thereby enabling you to concentrate primarily on strategic growth and ensuring unparalleled client success.

How Your Canvas Fingerprint Gets You Caught (And Why Random Noise Makes It Worse)

[Tanwydd] — Sat, 25 Apr 2026 08:54:12 +0000

You fixed the TLS fingerprint. You patched navigator.webdriver. Your User-Agent is perfect. And you're still getting blocked.

Chances are it's the Canvas.

What canvas fingerprinting actually is

Every browser renders graphics slightly differently. The GPU, the driver version, the OS font rendering engine, the antialiasing settings — all of these introduce tiny variations in how pixels end up on screen.

Canvas fingerprinting exploits this. The detection script draws something on an invisible canvas element — usually a mix of text, shapes and gradients — then reads back the pixel data with toDataURL() or getImageData(). The resulting string is hashed and becomes your fingerprint.

The variations are tiny. We're talking about differences at the level of individual pixel values, often invisible to the human eye. But they're consistent — the same browser on the same machine produces the same hash every time. And they're unique enough to identify you across sessions, across IPs, across proxies.

Your IP changes. Your canvas hash doesn't.

Why headless Chromium is obvious

Here's the specific problem with running Playwright or Puppeteer in headless mode: without a GPU pipeline, the browser falls back to software rendering.

Software rendering is deterministic and perfect. No GPU quirks, no driver variations, no antialiasing artifacts. Every headless Chromium instance on every machine produces an identical canvas output.

That's not how real browsers work. Real browsers are slightly imperfect in ways that are consistent per device. A perfectly identical canvas hash across thousands of sessions is a massive red flag.

The fix for this specific problem is --headless=new — Playwright's modern headless mode that preserves the full GPU pipeline:

context = await playwright.chromium.launch_persistent_context(
    headless=False,
    args=["--headless=new"],  # preserves GPU stack
)

But even with --headless=new, your canvas hash is still consistent across sessions. Which brings us to the noise problem.

Why random noise makes things worse

The obvious solution seems to be: add random noise to the canvas output on every render. Randomize the pixel values slightly so the hash changes.

This is wrong. And it's worse than doing nothing.

Here's why: real browsers produce a consistent canvas hash per device. The same machine always gives the same result. If your canvas hash changes on every page load, every request, every session — that's not how any real browser behaves. Detection systems don't just check what your hash is. They check whether it's stable.

A canvas hash that changes randomly is as obvious as navigator.webdriver = true. It's a different signal, but it's still a signal.

The right approach: deterministic per-session noise

What you want is noise that is:

Consistent within a session — the same hash for the same browser instance
Different across sessions — different profile directories produce different hashes
Realistic in magnitude — tiny variations, not wholesale pixel changes

The way to achieve this is a seeded pseudo-random number generator, where the seed is derived from something stable per profile — like the profile directory name.

import hashlib

def _session_seed(profile_dir_name: str) -> int:
    return int(hashlib.md5(profile_dir_name.encode()).hexdigest()[:8], 16) % (2**31)

Then use that seed to drive a simple LCG (Linear Congruential Generator) in JavaScript, and apply tiny pixel-level noise based on it:

// Seeded LCG — deterministic per session
let _seed = SESSION_SEED;
const _lcg = () => {
    _seed = (_seed * 1664525 + 1013904223) % 4294967296;
    return _seed / 4294967296;
};

// Apply noise without mutating the original canvas
const _applyNoise = (imgData) => {
    for (let i = 0; i < imgData.data.length; i += 4) {
        if (_lcg() < 0.05) {
            imgData.data[i] += (_lcg() > 0.5 ? 1 : -1);
        }
    }
};

The key detail: modify a copy of the canvas, not the original. If you mutate the original canvas, you break legitimate rendering on the page. The correct approach intercepts toDataURL(), getImageData() and toBlob(), draws to a temporary off-screen canvas, applies noise there, and returns the result.

const _origToDataURL = HTMLCanvasElement.prototype.toDataURL;
HTMLCanvasElement.prototype.toDataURL = function() {
    const ctx = this.getContext('2d');
    if (ctx) {
        const off = document.createElement('canvas');
        off.width = this.width;
        off.height = this.height;
        const octx = off.getContext('2d');
        octx.drawImage(this, 0, 0);
        const img = _origGetImageData.call(octx, 0, 0, off.width, off.height);
        _applyNoise(img);
        octx.putImageData(img, 0, 0);
        return _origToDataURL.apply(off, arguments);
    }
    return _origToDataURL.apply(this, arguments);
};

The toString() problem

There's a secondary issue that most canvas spoofing implementations miss: Function.prototype.toString().

When you replace a native browser function with your own JavaScript wrapper, any script that calls .toString() on that function sees JavaScript source code instead of function toDataURL() { [native code] }. That's detectable.

The fix is to maintain a registry of patched functions and override Function.prototype.toString to return the native code string for any function in that registry:

const _patchedFns = new WeakSet();

const _native = (fn, name) => {
    try {
        Object.defineProperty(fn, 'name', { value: name, configurable: true });
    } catch(_) {}
    _patchedFns.add(fn);
    return fn;
};

const _origFnToString = Function.prototype.toString;
Function.prototype.toString = _native(function() {
    if (_patchedFns.has(this)) {
        return `function ${this.name || ''}() { [native code] }`;
    }
    return _origFnToString.call(this);
}, 'toString');

Any function wrapped with _native() now looks indistinguishable from a browser built-in when inspected.

Canvas is one signal among many

Fixing canvas fingerprinting is necessary but not sufficient. Detection systems correlate multiple signals:

WebGL fingerprinting — same concept, different API. The GPU vendor string, renderer string, and the output of readPixels() all contribute to a fingerprint. The same deterministic noise approach applies.

AudioContext fingerprinting — the Web Audio API processes a signal through an oscillator and reads back the output. Again, tiny hardware-level variations create a unique hash. Tiny noise on getChannelData() output breaks this.

Font enumeration — document.fonts.check() and measureText() reveal which fonts are installed, which varies by OS. The browser's reported font list should match the OS implied by the User-Agent.

getBoundingClientRect() noise — font rendering affects element dimensions. Tiny noise on bounding rect values breaks font fingerprinting via layout measurement.

These signals are correlated. A Windows User-Agent with a Linux font list is suspicious. A Mac User-Agent with an NVIDIA GPU renderer is suspicious. Coherence across all signals matters as much as any individual fix.

The timing problem

One more thing that's easy to miss: your JavaScript wrappers add overhead. toDataURL() now does extra work — copying the canvas, applying noise, returning the result. That takes time.

Detection scripts measure how long canvas operations take. A toDataURL() call that takes 3x longer than expected is a signal.

The fix is to add a small amount of noise to performance.now() so timing measurements are slightly unpredictable:

const _origPerfNow = performance.now.bind(performance);
performance.now = _native(function() {
    return _origPerfNow() + (_lcg() - 0.5) * 0.2;
}, 'now');

±0.1ms of jitter is enough to mask the overhead of your wrappers without being detectable itself.

Next: mouse movement, typing speed, and why behavioral fingerprinting is harder to fake than canvas — and what that looks like in code.

Build a Daily Horoscope Feature in Python in 10 Minutes (with DivineAPI)

Karan Desai — Sat, 25 Apr 2026 08:49:49 +0000

The problem

A client wanted daily horoscopes on their wellness app. All 12 signs, refreshed daily, English first with Hindi coming in phase two. The deadline was end of week. I did not want to spend three days reading documentation for an API I had never used.

Here is the stack I ended up with, and the working Python you can drop into a project today.

What I used

DivineAPI for the horoscope endpoint
Python 3.11
The requests library (pip install requests)
About 10 minutes of actual work

Step 1: Get your API key

Sign up at divineapi.com, 14-day free trial, you will need a credit card to register. Once you are in the dashboard, grab your API key. Two things they hand you that you will need:

The api_key value (goes in the request body)
An Authorization token (goes in the header as Bearer {token})

Yes, two separate auth bits. I did not catch this on first read and spent ten minutes wondering why I was getting 401s. More on that below.

Step 2: Your first call

Here is the smallest working request for the daily horoscope endpoint:

import requests

url = "https://astroapi-5.divineapi.com/api/v5/daily-horoscope"

headers = {
    "Authorization": "Bearer YOUR_AUTH_TOKEN"
}

payload = {
    "api_key": "YOUR_API_KEY",
    "sign": "leo",
    "h_day": "today",        # also accepts "yesterday" or "tomorrow"
    "tzone": 5.5              # IST. timezone offset as a float, not a string
}

response = requests.post(url, headers=headers, json=payload)
data = response.json()

# the prediction is split into 5 life areas + a "luck" array
print(data["data"]["prediction"]["personal"])

That is it. One POST, one JSON response, daily horoscope for Leo.

A few non-obvious bits the docs are right about but easy to miss:

The endpoint is on astroapi-5.divineapi.com, not the apex divineapi.com. They run different astro services on numbered subdomains.
The day parameter is named h_day, not day. I assumed day and got a 422.
tzone is a Float, 5.5 for IST. Not a string. Not "Asia/Kolkata". Not 5. I learned this the hard way (see "the part where I got stuck").
For non-English you swap the host to astroapi-5-translator.divineapi.com and add a lan field. They support 25 languages including Indonesian and Greek (the language list got expanded in April 2026).

What comes back

Here is the actual response shape, trimmed for readability:

{
  "success": 1,
  "data": {
    "sign": "Leo",
    "date": "2026-04-23",
    "prediction": {
      "personal": "Your personal life is full of vibrant energy today...",
      "health": "Pay attention to physical wellness today, Leo...",
      "profession": "Today, you may find yourself inspired at work, Leo...",
      "emotions": "Emotionally, you may feel heightened sensitivity, Leo...",
      "travel": "Adventure calls today, Leo! Whether it's a spontaneous day trip...",
      "luck": [
        "Colors of the day : Gold, Orange",
        "Lucky Numbers of the day : 4, 9, 13",
        "Lucky Alphabets you will be in sync with : L, S",
        "Cosmic Tip : Beware of impulsive actions; stay grounded and focused.",
        "Tips for Singles : Express your interests confidently...",
        "Tips for Couples : Share your dreams; strengthen your bond..."
      ]
    },
    "special": {
      "lucky_color_codes": ["#FFD700", "#FFA500"],
      "horoscope_percentage": {
        "personal": 85, "health": 75, "profession": 80, "emotions": 78
      }
    }
  }
}

Two things worth noticing. The prediction is broken into five life areas (personal, health, profession, emotions, travel) plus a luck array of one-liners. So you can render the full daily horoscope, or just the personal text, or just the lucky numbers as a widget. Useful flexibility. Second, special.lucky_color_codes gives you actual hex codes, which means you can theme the daily card with the lucky colour without writing a sign-to-colour map yourself.

Average response times in my testing came back around 74ms over a few hundred requests, which tracks with their published "from 72ms" claim. Fast enough that you do not need a loading state for a single sign.

Step 3: All 12 signs in one loop

For a daily-refresh job that fills a database or warms a cache, you want every sign in one go. Wrap the call in a function and loop:

import requests

URL = "https://astroapi-5.divineapi.com/api/v5/daily-horoscope"
HEADERS = {"Authorization": "Bearer YOUR_AUTH_TOKEN"}
API_KEY = "YOUR_API_KEY"

SIGNS = [
    "aries", "taurus", "gemini", "cancer", "leo", "virgo",
    "libra", "scorpio", "sagittarius", "capricorn", "aquarius", "pisces"
]

def fetch_daily(sign):
    payload = {
        "api_key": API_KEY,
        "sign": sign,
        "h_day": "today",
        "tzone": 5.5
    }
    r = requests.post(URL, headers=HEADERS, json=payload, timeout=10)
    r.raise_for_status()
    return r.json()["data"]

results = {sign: fetch_daily(sign) for sign in SIGNS}

# now `results["leo"]["prediction"]["personal"]` etc.
print(f"Fetched {len(results)} sign predictions")

Twelve calls, completes in under a second on a decent connection. If you are serving these to users live, cache the response per-sign per-day. The data only changes once every 24 hours. There is no good reason to hit the API on every page load.

Step 4: Add weekly and monthly horoscopes

Same auth, same response shape, different endpoint and one parameter name change:

WEEKLY_URL  = "https://astroapi-5.divineapi.com/api/v5/weekly-horoscope"
MONTHLY_URL = "https://astroapi-5.divineapi.com/api/v5/monthly-horoscope"

def fetch_weekly(sign):
    payload = {
        "api_key": API_KEY,
        "sign": sign,
        "week": "current",     # also "prev" or "next"
        "tzone": 5.5
    }
    r = requests.post(WEEKLY_URL, headers=HEADERS, json=payload, timeout=10)
    return r.json()["data"]

def fetch_monthly(sign):
    payload = {
        "api_key": API_KEY,
        "sign": sign,
        "month": "current",    # also "prev" or "next"
        "tzone": 5.5
    }
    r = requests.post(MONTHLY_URL, headers=HEADERS, json=payload, timeout=10)
    return r.json()["data"]

Worth knowing: DivineAPI ships 6 horoscope endpoints total under the Horoscope and Tarot category. Daily, Weekly, Monthly, Yearly, Chinese Horoscope, and Numerology Horoscope. So if you ever want to add yearly forecasts or Chinese zodiac alongside the Western ones, the same API key covers all of them.

The part where I got stuck

Two stupid bugs cost me about 25 minutes total. Both worth flagging because they will cost you time too if nobody warns you.

Bug 1: "day" vs "h_day". I assumed the parameter would be called day because that is what I would call it. The actual name is h_day. The API responded with a 422 and a perfectly clear error message saying the field was missing. I looked at my code, saw day, and somehow my brain kept reading it as h_day. Three minutes of "but it IS there, what is wrong with this thing". Lesson: when an API says a field is missing and you swear it is there, check the name character by character.

Bug 2: timezone as a string. I sent "tzone": "5.5" because I was building the payload from form input and forgot to cast. The API accepted it, returned a 200, and gave me back a horoscope. Looked fine. Except the date in the response was off by one for a chunk of users, depending on what time of day they hit the endpoint. Took a while to pin down. The API expects tzone as a Float. Send a string and it silently degrades the date calculation. Cast your inputs.

If you are integrating this into a Django or Flask form, do the cast at the serializer layer and write a test for it.

What I'd do differently

Three things, in order of how much pain they would have saved.

Cache aggressively. I should have set up a Redis cache from the start. The horoscope data is per-sign per-day. Twelve API calls per day total, the rest served from cache. Saves the API budget and removes the latency entirely.
Centralise the auth headers. I scattered HEADERS = {...} across three files before I refactored. A small wrapper function or a requests.Session would have been better from day one.
Don't roll your own retry logic. Use tenacity or requests-retry-session for the 5xx case. I wrote a half-baked retry loop, immediately regretted it, and ripped it out for the library.

Wrapping up

That is a working horoscope feature in about 10 minutes of code. You can extend it to weekly, monthly, yearly, Chinese, or numerology horoscopes with the same API key. They have a Postman collection at the docs website which is genuinely useful for poking at the response before you write any code. Full docs at developers.divineapi.com.

DivineAPI has a 14-day free trial if you want to try this yourself. If you ship something with it, drop a comment. Always curious what people end up building.

Claude Code: Hooks, Subagents, and Skills — Complete Guide

Owen — Sat, 25 Apr 2026 08:45:11 +0000

Claude Code: Hooks, Subagents, and Skills — Complete Guide

Claude Code offers three extensibility layers: hooks for lifecycle automation, subagents for parallel task delegation, and skills for reusable prompt templates. This guide explains each mechanism, when to apply which, and how to combine them effectively.

What This Guide Covers

Hooks, subagents, and skills transform Claude Code from a conversational tool into a programmable AI engineering platform. For foundational setup, reference the configuration guide and model selection documentation.

Hooks: Deterministic Control Over Claude Code

Hooks are event-driven scripts executing when something happens in Claude Code. Unlike prompts relying on model interpretation, hooks run deterministic code incapable of hallucination.

Why Hooks Matter

Without hooks, every safeguard depends on the model understanding instructions. With hooks, rules enforce at the system level. Block dangerous commands before execution. Inject project context automatically. Log every tool call for audit purposes.

Hook Types and What They Do

Type	What It Runs	Best For
`command`	Shell script receiving JSON on stdin	Blocking dangerous commands, local validation
`http`	HTTP POST endpoint	Centralized policy enforcement, remote logging
`mcp_tool`	Connected MCP server tool	Integration with external security scanners
`prompt`	Single-turn LLM evaluation	Semantic validation ("does this look like a secret?")
`agent`	Subagent using tools to verify	Complex multi-step validation before approval

The 25 Lifecycle Events

Hooks fire at 25 distinct lifecycle points. Blocking-capable events include:

UserPromptSubmit — Fires when you submit a prompt. Can block or modify the prompt before Claude sees it.
PreToolUse — Fires before any tool executes. The primary security checkpoint.
PermissionRequest — Fires when Claude asks for permission. Can auto-approve or deny.
Stop / SubagentStop — Fires when Claude or a subagent finishes. Can force continuation.
PreCompact — Fires before context compaction. Can back up transcripts.

Informational events cannot block but can log or notify:

SessionStart / SessionEnd — Session lifecycle. Load context on start, clean up on end.
PostToolUse / PostToolUseFailure — Tool completion or failure. Log results, run linters.
SubagentStart — Subagent spawned. Track agent orchestration.
Notification — Claude sends a notification. Route to Slack, trigger TTS.

Exit Code Behavior

Exit Code	Meaning
`0`	Success. stdout parsed for JSON decisions.
`2`	Blocking error. stderr fed to Claude; action blocked.
`1` or other	Non-blocking error. First line of stderr shown; execution continues.

Example: Block Dangerous Commands with PreToolUse

Create .claude/hooks/block-rm.sh:

#!/bin/bash
COMMAND=$(jq -r '.tool_input.command')

if echo "$COMMAND" | grep -q 'rm -rf'; then
  jq -n '{
    hookSpecificOutput: {
      hookEventName: "PreToolUse",
      permissionDecision: "deny",
      permissionDecisionReason: "Destructive command blocked by hook"
    }
  }'
  exit 2
else
  exit 0
fi

Configure in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "if": "Bash(rm *)",
            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/block-rm.sh"
          }
        ]
      }
    ]
  }
}

Now any rm -rf command is blocked before execution, with the denial reason shown to Claude.

Example: Auto-Inject Project Context on SessionStart

#!/bin/bash
# .claude/hooks/session-start.sh
if [ -f "$PWD/CLAUDE.md" ]; then
  echo "Loaded project context from CLAUDE.md"
fi
if [ -f "$PWD/.env.example" ]; then
  echo "Environment template available at .env.example"
fi
exit 0

This runs every time Claude Code starts in a directory, surfacing relevant context automatically.

Hook Scopes

Location	Scope
`~/.claude/settings.json`	All projects
`.claude/settings.json`	Single project
`.claude/settings.local.json`	Single project, not shared
Skill/agent frontmatter	Component lifetime

Project-level hooks are ideal for team-shared policies. Personal hooks in ~/.claude/ apply everywhere.

Subagents: Parallel Workers with Isolated Context

Subagents are specialized AI instances handling tasks in their own context window. When a subagent runs, its verbose output — file searches, log dumps, multi-step reasoning — stays isolated. Only the summary returns to your main conversation.

Built-in Subagents

Subagent	Model	Tools	Purpose
Explore	Haiku	Read-only	Fast codebase search and analysis
Plan	Inherits	Read-only	Research for plan mode
General-purpose	Inherits	All tools	Complex multi-step tasks

Claude delegates automatically based on task type. You can also invoke explicitly with @agent-name or claude --agent <name>.

When to Use Subagents

Use subagents when:

A task produces verbose output you do not need in main context
You want to enforce tool restrictions (e.g., read-only review)
You need parallel research on independent topics
The work is self-contained and can return a summary

Use the main conversation when:

The task needs frequent back-and-forth refinement
Multiple phases share significant context
Latency matters (subagents start fresh)

Creating a Custom Subagent

Subagents are Markdown files with YAML frontmatter. Save to .claude/agents/ (project) or ~/.claude/agents/ (personal):

---
name: code-reviewer
description: Expert code review specialist. Proactively reviews code for quality, security, and maintainability. Use immediately after writing or modifying code.
tools: Read, Grep, Glob, Bash
model: sonnet
---

You are a senior code reviewer ensuring high standards of code quality and security.

When invoked:
1. Run git diff to see recent changes
2. Focus on modified files
3. Begin review immediately

Review checklist:
- Code is clear and readable
- Functions and variables are well-named
- No duplicated code
- Proper error handling
- No exposed secrets or API keys
- Input validation implemented
- Good test coverage
- Performance considerations addressed

Provide feedback organized by priority:
- Critical issues (must fix)
- Warnings (should fix)
- Suggestions (consider improving)

Include specific examples of how to fix issues.

Invoke with: Use the code-reviewer agent to review my auth changes

Or guarantee delegation with @-mention: @"code-reviewer (agent)" look at the auth changes

Subagent Configuration Options

Field	Purpose
`tools`	Allowlist of tools the subagent can use
`disallowedTools`	Denylist (e.g., `Write, Edit` for read-only agents)
`model`	`sonnet`, `opus`, `haiku`, `inherit`, or full model ID
`permissionMode`	`default`, `acceptEdits`, `auto`, `dontAsk`, `bypassPermissions`, `plan`
`skills`	Preload skill content into subagent context
`mcpServers`	MCP servers scoped to this subagent
`hooks`	Lifecycle hooks scoped to this subagent
`memory`	Persistent memory: `user`, `project`, or `local`
`isolation`	`worktree` for git branch isolation
`maxTurns`	Maximum agentic turns before stopping

Preloading Skills into Subagents

Subagents do not inherit parent skills. Preload explicitly:

---
name: api-developer
description: Implement API endpoints following team conventions
skills:
  - api-conventions
  - error-handling-patterns
---

Implement API endpoints. Follow the conventions and patterns from the preloaded skills.

The full skill content is injected at startup, not just made available.

Forked Subagents (Experimental)

Forks inherit the full conversation history instead of starting fresh. Use them when a named subagent would need too much background context.

Enable: CLAUDE_CODE_FORK_SUBAGENT=1

Spawn: /fork draft unit tests for the parser changes

Forks run in the background while you continue working. Results arrive as messages when complete.

Parallel Research Pattern

Request: "Research the authentication, database, and API modules in parallel using separate subagents"

Each subagent explores independently. Claude synthesizes the findings. This works best when research paths do not depend on each other.

Skills: Reusable Prompts and Workflows

Skills extend what Claude can do by packaging instructions into invocable commands. Create a skill when you keep pasting the same playbook into chat.

Skills vs. CLAUDE.md

Aspect	CLAUDE.md	Skills
Loads	Automatically on session start	Only when invoked
Best for	Project conventions, permanent context	Procedures, playbooks, workflows
Cost	Always in context	Only when used

Unlike CLAUDE.md content, a skill's body loads only when invoked, so long reference material costs almost nothing until needed.

Creating Your First Skill

mkdir -p ~/.claude/skills/explain-code

Create ~/.claude/skills/explain-code/SKILL.md:

---
name: explain-code
description: Explains code with visual diagrams and analogies. Use when explaining how code works, teaching about a codebase, or when the user asks "how does this work?"
---

When explaining code, always include:

1. **Start with an analogy**: Compare the code to something from everyday life
2. **Draw a diagram**: Use ASCII art to show the flow, structure, or relationships
3. **Walk through the code**: Explain step-by-step what happens
4. **Highlight a gotcha**: What's a common mistake or misconception?

Keep explanations conversational. For complex concepts, use multiple analogies.

Invoke automatically: How does this code work?

Invoke directly: /explain-code src/auth/login.ts

Skill Frontmatter Reference

Field	Purpose
`name`	Display name; becomes the `/slash-command`
`description`	When Claude should use the skill automatically
`disable-model-invocation`	Set `true` to prevent auto-loading (for dangerous ops like deploy)
`user-invocable`	Set `false` to hide from `/` menu (background knowledge only)
`allowed-tools`	Tools Claude can use without asking permission when skill is active
`context`	Set `fork` to run in isolated subagent
`agent`	Which subagent type to use with `context: fork`
`model` / `effort`	Override model or effort level when skill is active
`paths`	Glob patterns limiting when skill auto-activates

Dynamic Context Injection

The !`command` syntax runs shell commands before the skill content is sent to Claude:

---
name: pr-summary
description: Summarize changes in a pull request
context: fork
agent: Explore
allowed-tools: Bash(gh *)
---

## Pull request context
- PR diff: !`gh pr diff`
- PR comments: !`gh pr view --comments`
- Changed files: !`gh pr diff --name-only`

## Your task
Summarize this pull request...

Commands execute immediately; Claude receives only the output. For multi-line commands, use `! fenced blocks.

Skill Directory Structure

`plaintext my-skill/ ├── SKILL.md # Main instructions (required) ├── template.md # Template for Claude to fill in ├── examples/ │ └── sample.md # Example output └── scripts/ └── validate.sh # Script Claude can execute `

Reference supporting files from SKILL.md so Claude knows what they contain and when to load them.

Where Skills Live

Location	Path	Scope
Enterprise	Managed settings	Organization-wide
Personal	`~/.claude/skills/<name>/SKILL.md`	All your projects
Project	`.claude/skills/<name>/SKILL.md`	This project only
Plugin	`<plugin>/skills/<name>/SKILL.md`	Where plugin is enabled

Higher-priority locations win: enterprise > personal > project. Plugin skills use plugin-name:skill-name namespace.

Bundled Skills

Claude Code includes built-in skills available in every session:

/simplify — Simplify complex code or explanations
/debug — Systematic debugging workflow
/batch — Process multiple items efficiently
/loop — Iterate on a task until complete
/claude-api — Reference for Claude API patterns

These are prompt-based, not hardcoded. They give Claude a detailed playbook and let it orchestrate the work.

Combining Hooks, Subagents, and Skills

The three features compose together. Here is a production-ready setup:

Example: Secure Code Review Pipeline

Skill (~/.claude/skills/secure-review/SKILL.md):

`markdown

name: secure-review
description: Security-focused code review. Use when reviewing authentication, authorization, or data handling code.
context: fork
agent: Explore

disable-model-invocation: true

Perform a security-focused code review:

Check for hardcoded secrets, API keys, or credentials
Verify input validation and sanitization
Review authentication and authorization logic
Check for SQL injection, XSS, and injection vulnerabilities
Verify error handling does not leak sensitive information
Check file upload and path traversal protections

Report findings with severity levels and specific file references.
`

Hook (.claude/settings.json):

`json { "hooks": { "PostToolUse": [ { "matcher": "Edit|Write", "hooks": [ { "type": "command", "command": "./scripts/run-security-linter.sh" } ] } ] } } `

Subagent (.claude/agents/security-reviewer.md):

`markdown

name: security-reviewer
description: Security review specialist for auth and data handling code
tools: Read, Grep, Glob, Bash

disallowedTools: Edit, Write

You are a security-focused code reviewer. Focus on:

Authentication and authorization flaws
Input validation gaps
Secret leakage
Injection vulnerabilities

Never modify code. Only report findings.
`

Usage: After editing auth code, run /secure-review or ask Claude to have the security-reviewer agent check these changes. The PostToolUse hook runs the security linter on every file edit automatically.

Practical Patterns

Pattern 1: Context Preservation

Use subagents for operations producing large outputs:

"Use a subagent to run the test suite and report only the failing tests with their error messages"

The full test output stays in the subagent's context. You get only the actionable summary.

Pattern 2: Tool Restriction

Limit what subagents can do for safety:

`markdown

name: db-reader
description: Execute read-only database queries
tools: Bash
hooks:
PreToolUse:
- matcher: "Bash"
hooks:
- type: command

command: "./scripts/validate-readonly-query.sh"

The hook blocks any SQL write operation before it executes.

Pattern 3: Model Routing

Route different tasks to different models for cost optimization:

`markdown

name: quick-classifier
description: Classify incoming requests by type and complexity

model: haiku

Classify this request as: simple, complex, or research-heavy.
`

Haiku is fast and cheap for classification. Route complex tasks to Sonnet or Opus.

Pattern 4: Persistent Memory

Enable cross-session learning for subagents:

`markdown

name: codebase-architect
description: Maintains architectural knowledge of the codebase

memory: project

Update your agent memory as you discover codepaths, patterns, library locations, and key architectural decisions.
`

The subagent accumulates knowledge in .claude/agent-memory/codebase-architect/ across conversations.

Accessing Claude Code Through OfoxAI

Claude Code works with any Anthropic-compatible API endpoint. OfoxAI provides full protocol support including extended thinking and cache_control.

Configuration:

Request URL: https://api.ofox.ai/anthropic
API Key: Your OfoxAI key from app.ofox.ai

For setup instructions, see the Claude Code configuration guide. For model comparisons, see Claude Opus 4.7 API review and best AI model for agents 2026.

Summary

Feature	What It Does	Use When
Hooks	Deterministic lifecycle scripts	Security, logging, context injection, blocking
Subagents	Isolated AI workers	Parallel tasks, verbose output isolation, tool restriction
Skills	Reusable prompt templates	Repeatable workflows, conventions, team knowledge

Start with skills — they are the easiest to create and provide immediate value. Add hooks when you need deterministic enforcement. Use subagents when parallel work or context isolation matters.

The teams getting the most from Claude Code treat it as a programmable platform, not just a chat interface. Hooks, subagents, and skills are the tools that make that transition possible.

Originally published on ofox.ai/blog.

DEV Community

The Machine Learning Lifecycle: 10 Steps From Problem to Production (And Why Most Projects Fail at Step 3)

The Big Picture

1. Problem Definition — "What are we actually solving?"

2. Data Collection — "Do we have enough?"

3. Data Cleaning and Preprocessing — "Garbage in, garbage out"

Handling Missing Values

Removing Duplicates

Fixing Inconsistent Data

Handling Outliers

Data Type Conversions

Normalization and Scaling

4. Exploratory Data Analysis (EDA) — "What does the data actually look like?"

5. Feature Engineering and Selection — "Good features > fancy models"

Feature Engineering (Creating)

Feature Selection (Removing)

6. Model Selection — "Choose the right tool for the job"

7. Model Training — "The model learns about the data"

8. Model Evaluation and Tuning — "How is your model doing?"

Evaluation Metrics

Hyperparameter Tuning

Dealing with Problems

9. Model Deployment — "Integrate model to the real world"

10. Monitoring and Maintenance — "Keep model healthy"

What to Monitor

When to Retrain

Best Practices

Summary — The Key Takeaways

Learn Kubernetes the Manga Way

Learn Kubernetes the Manga Way - Payhip

AI actually made my tutorial addiction worse

AI #WebDev #DeveloperLife #TutorialHell #LearningToCode

Plausible vs Fathom Analytics: Which Fits in 2026?

What Plausible and Fathom actually optimize for

Privacy, compliance, and the “no cookies” reality

Tracking depth: pageviews vs events (and when you outgrow them)

Implementation: a minimal event you can ship today

Choosing between Plausible and Fathom (and what to pair them with)

N+1 Query Detection and Prevention in Laravel Production Apps

Understanding the N+1 Problem

Prevention: Laravel's preventLazyLoading

Handling Violations in Production

When preventLazyLoading Gets in the Way

Eager Loading Patterns

Basic Eager Loading

Nested Eager Loading

Constrained Eager Loading

Conditional Eager Loading

Eager Loading on Existing Collections

Default Eager Loading on Models

Detecting N+1 in Existing Code

Query Counting in Tests

Using Laravel Debugbar

Using Telescope's Query Tab

Common N+1 Traps

Trap 1: Blade Components That Access Relationships

Trap 2: API Resources with Nested Relationships

Trap 3: Accessors That Touch Relationships

Trap 4: Queued Jobs Processing Collections

Monitoring in Production

Laravel Pulse

Deploynix Server Monitoring

Custom Middleware for Query Counting

Conclusion

Advanced PostgreSQL: JSONB, partial indexes and partitioning

JSONB: storing flexible schemas without sacrificing performance

json vs jsonb — always use jsonb

Essential operators

Adding the column and GIN index

When not to use JSONB

Partial indexes: only index what matters

The problem with full indexes

The solution: partial index on active rows only

Other common use cases

EXPLAIN ANALYZE: reading between the lines

The full command

Decoding the plan

Pattern to watch for: the expensive Sort node

Date partitioning: the solution for massive tables

Creating the partitioned table

`json` vs `jsonb` — always use `jsonb`