A simple beginner-friendly guide to Advanced Product Quality Planning (APQP) in the automotive industry with examples, flow diagrams, and easy explanations.

Imagine you want to build the best toy car ever for your friends.

Would you start cutting plastic and gluing wheels together immediately?

Probably not.

You would first:

• Think about what the toy should look like
• Draw a picture
• Gather the right materials
• Build one toy
• Test if it works
• Fix any problems
• Then make lots of toys

That is exactly how car companies build real vehicles and their parts.

This process is called APQP.

What Does APQP Stand For?

APQP = Advanced Product Quality Planning

Don't let the long name scare you.

It simply means:

A step-by-step way to make sure a product is good before making thousands or millions of them.

Instead of fixing mistakes after production, companies try to prevent mistakes before production starts.

Why Do We Need APQP?

Imagine making 100,000 cars.

If one tiny bolt is installed incorrectly...

You don't fix one car.

You fix 100,000 cars.

That could cost millions of dollars.

Instead, companies follow this simple idea:

Think first. Test first. Build later.

A Simple Toy Example

Imagine your teacher says:

"Tomorrow, everyone must bring one paper airplane."

There are two ways to do it.

❌ Without APQP

Make airplane quickly

↓

It doesn't fly

↓

Teacher is unhappy

✅ With APQP

Think

↓

Choose the best paper

↓

Practice folding

↓

Test flying

↓

Improve

↓

Make the final airplane

The second approach gives everyone a much better airplane.

That's APQP.

Real Automotive Example

Suppose a company wants to manufacture a car door handle.

Without APQP

Design

↓

Manufacture 50,000 handles

↓

Oops!

The handle breaks easily.

↓

Replace every handle

This is expensive and wastes time.

With APQP

Design

↓

Build prototype

↓

Test

↓

Improve

↓

Test again

↓

Approve

↓

Mass production

Now customers receive reliable products.

Think of Building a House

You don't start by building the roof.

You follow a plan.

Idea

↓

Drawing

↓

Buy materials

↓

Build

↓

Inspect

↓

Move in

Building cars works the same way.

The 5 Phases of APQP

Every new automotive product follows five simple phases.

Phase 1 — Plan

First, understand what the customer wants.

Questions include:

• What product are we making?
• How strong should it be?
• How much should it cost?
• When should it be ready?

Example

A customer says:

"I need a steering wheel."

The team asks:

• Leather or plastic?
• Airbag included?
• Buttons required?
• What size?

This phase is all about planning.

Phase 2 — Product Design

Now engineers design the product.

Example:

Steering Wheel

Diameter: 380 mm

Leather Cover

Airbag

Control Buttons

Weight: 1.8 kg

Now everyone knows exactly what needs to be built.

Phase 3 — Process Design

The question changes.

Instead of asking:

"How should we design it?"

We ask:

"How will we manufacture it?"

Example production flow:

Raw Material

↓

Machine

↓

Assembly

↓

Painting

↓

Inspection

↓

Packaging

This phase plans how the factory will produce the product.

Phase 4 — Product & Process Validation

Now it's time to test everything.

Questions include:

• Does the product work?
• Is it safe?
• Can the factory consistently make good parts?

Example:

Manufacture 300 steering wheels.

Test every one.

If everything passes...

Production can begin.

If not...

Fix the problem and test again.

Phase 5 — Production

Everything has been approved.

Now the factory starts producing thousands—or even millions—of parts.

Even during production, quality is checked continuously.

The Complete APQP Flow

Customer Requirement

        │

        ▼

Phase 1
Planning

        │

        ▼

Phase 2
Product Design

        │

        ▼

Phase 3
Process Design

        │

        ▼

Phase 4
Testing & Validation

        │

        ▼

Problems Found?

   Yes ────────────────┐
        │              │
        ▼              │
   Improve Design      │
        │              │
        └──────► Test Again

No

↓

Phase 5

Mass Production

Another Easy Example: Baking Cookies 🍪

Imagine your mom wants to bake cookies.

Phase 1

Choose a recipe.

Phase 2

Select ingredients.

Flour

Sugar

Butter

Chocolate Chips

Phase 3

Plan the baking process.

Mix

↓

Shape

↓

Bake

↓

Cool

↓

Pack

Phase 4

Taste one cookie.

Too hard?

Add more butter.

Too soft?

Bake longer.

Phase 5

Bake 500 cookies using the improved recipe.

That's APQP in everyday life.

Another Automotive Example

Suppose a company is designing a brake pedal.

Without APQP

Design

↓

Manufacture

↓

Customer uses pedal

↓

Pedal breaks

A dangerous situation.

With APQP

Customer Requirement

↓

Design

↓

Computer Simulation

↓

Prototype

↓

Strength Testing

↓

Improve

↓

Mass Production

Now the brake pedal is much safer.

APQP Is Like Preparing for an Exam

Imagine you have a big school exam.

What would you do?

Study

↓

Practice

↓

Find Mistakes

↓

Practice Again

↓

Take Exam

↓

Pass

You don't wait until the exam to discover what you don't know.

Car companies think the same way.

They don't wait until customers find problems.

They discover and fix problems first.

Who Works During APQP?

Many different teams work together.

Everyone collaborates from the beginning.

An Easy Way to Remember APQP

Think of APQP as this simple journey:

Think

↓

Plan

↓

Design

↓

Build

↓

Test

↓

Improve

↓

Produce

Or even shorter:

Think → Plan → Test → Improve → Produce

Key Benefits of APQP

Using APQP helps companies:

• Reduce mistakes
• Improve product quality
• Lower manufacturing costs
• Prevent customer complaints
• Improve safety
• Deliver products on time
• Build consistent products every time

Final Summary

APQP is simply a smart planning process used by automotive companies.

Instead of rushing into production, companies:

1. Understand customer needs
2. Design the product
3. Design the manufacturing process
4. Test everything carefully
5. Start mass production only after everything works

Just like practicing before a school play or testing a recipe before serving guests, APQP helps manufacturers build safer, better, and more reliable products.

Remember This One Sentence

APQP is like practicing before the final performance—you plan everything, test everything, fix every mistake, and only then build thousands of products with confidence.

Learn how to design, analyze, and improve a business using the Business Model Canvas (BMC) framework. This guide explains each of the nine building blocks with a practical QMS (Quality Management System) Issue Management Platform example.

What is a Business Model Canvas?

A Business Model Canvas (BMC) is a one-page strategic framework that helps organizations visualize how a business creates, delivers, and captures value.

Instead of writing a lengthy business plan, the Business Model Canvas allows founders, product managers, consultants, and business leaders to quickly understand:

• Who are the customers?
• What value is being delivered?
• How is value delivered?
• How does the business generate revenue?
• What resources and partners are required?

The framework was introduced by Alexander Osterwalder and has become one of the most widely used business planning tools worldwide.

Why Use a Business Model Canvas?

Business Model Canvas helps organizations:

• Validate business ideas quickly
• Align teams around a common business vision
• Identify business risks
• Improve existing business models
• Discover new revenue opportunities
• Simplify strategic planning
• Present ideas to investors and stakeholders

The 9 Building Blocks of Business Model Canvas

The Business Model Canvas consists of nine interconnected building blocks.

Reference - https://en.wikipedia.org/wiki/Business_model_canvas

1. Customer Segments

What is it?

Customer Segments define who your customers are.

Every business serves one or more customer groups. Understanding them is the first step toward building a successful product.

Ask yourself:

• Who are our customers?
• Who gets the most value?
• Who pays?
• Who uses the product?

Example: QMS Issue Management Platform

Primary customers include:

• OEM Quality Engineers
• OEM Supplier Quality Teams
• Supplier Engineers
• Supplier Managers
• Supplier Quality Teams

Why it matters

Without clearly identifying customer segments, it becomes difficult to create products that solve real problems.

2. Value Proposition

What is it?

The Value Proposition explains why customers choose your product instead of alternatives.

It answers:

What problem are you solving?

Examples include:

• Saving time
• Reducing cost
• Improving quality
• Increasing productivity
• Better customer experience

Example: QMS Issue Management Platform

For OEMs

• Real-time supplier issue tracking
• Faster issue resolution
• Centralized quality management
• Better supplier performance visibility

For Suppliers

• Receive issues instantly
• Track corrective actions
• Automated reminders
• Better collaboration with OEMs

Why it matters

Customers don't buy software.

They buy solutions to their problems.

3. Key Resources

What is it?

Key Resources are the assets required to deliver your value proposition.

They can include:

• Technology
• Software
• Infrastructure
• Employees
• Brand
• Data
• Intellectual Property

Example: QMS Issue Management Platform

Key resources include:

• Cloud Platform
• Issue Tracking Software
• Notification Engine
• Reporting Dashboard
• Supplier Network
• Customer Database
• Security Infrastructure
• Development Team

Why it matters

Without the right resources, the business cannot operate effectively.

4. Key Partners

What is it?

Key Partners are organizations that help your business succeed.

Examples:

• Suppliers
• Technology providers
• Cloud vendors
• Consultants
• Payment providers

Example: QMS Issue Management Platform

Partners include:

• OEM Companies
• Supplier Organizations
• Cloud Infrastructure Providers
• Email & Notification Services
• Authentication Providers
• Technology Vendors
• Compliance Consultants

Why it matters

Strategic partnerships reduce costs and increase business capabilities.

5. Key Activities

What is it?

Key Activities describe the critical work required to operate the business.

Examples include:

• Software development
• Marketing
• Customer support
• Sales
• Operations

Example: QMS Issue Management Platform

Important activities:

• Build platform
• Maintain software
• Deploy new features
• Monitor system uptime
• Customer onboarding
• Technical support
• Security updates
• Regulatory compliance
• Platform monitoring

Why it matters

Activities transform resources into customer value.

6. Customer Relationships

What is it?

Customer Relationships describe how you interact with customers throughout their journey.

Examples include:

• Self-service
• Personal support
• Dedicated account manager
• Community forums
• Knowledge base

Example: QMS Issue Management Platform

Relationship methods:

• 99.9% System Availability
• Automated Notifications
• Email Alerts
• Help Desk
• Knowledge Base
• Training Sessions
• Customer Success Team
• SLA-based Support

Why it matters

Strong customer relationships improve retention and satisfaction.

7. Revenue Streams

What is it?

Revenue Streams explain how the business earns money.

Possible models include:

• Subscription
• Licensing
• Usage-based pricing
• Premium support
• Consulting
• Advertising

Example: QMS Issue Management Platform

Revenue sources:

• Subscription per organization
• Number of users
• Number of issues managed
• Data storage usage
• Premium analytics
• Enterprise support
• API integrations
• Training services

Why it matters

Revenue determines business sustainability.

8. Cost Structure

What is it?

Cost Structure identifies where money is spent.

Typical costs include:

• Salaries
• Cloud hosting
• Marketing
• Licenses
• Infrastructure
• Research
• Customer Support

Example: QMS Issue Management Platform

Major costs:

• Employee Salaries
• Cloud Infrastructure
• Technology Development
• Software Licenses
• Marketing
• Customer Support
• Security Compliance
• Research & Development

Why it matters

Understanding costs helps improve profitability.

9. Channels

What is it?

Channels define how customers discover, purchase, and use your product.

Examples include:

• Website
• Mobile App
• Web Application
• Social Media
• Sales Team
• Partners

Example: QMS Issue Management Platform

Customer channels:

• Web Application
• Mobile Application
• Email
• Sales Team
• Customer Portal
• Social Media
• Product Demonstrations
• Online Documentation

Why it matters

Even a great product needs effective channels to reach customers.

Business Model Canvas Example

QMS Issue Management Platform

Tips for Creating an Effective Business Model Canvas

✅ Focus on solving customer problems.

✅ Keep customer segments specific.

✅ Clearly define your unique value proposition.

✅ Identify measurable revenue streams.

✅ Build strategic partnerships.

✅ Regularly revisit and update your canvas.

✅ Validate assumptions through customer feedback.

Common Mistakes

• Trying to serve everyone
• Weak value proposition
• Ignoring customer relationships
• Missing revenue strategy
• Underestimating operating costs
• Not identifying key partners
• Treating the canvas as a one-time activity

Benefits of Using Business Model Canvas

• Simple one-page business overview
• Faster strategic planning
• Easier stakeholder communication
• Better product-market fit
• Improved innovation
• Faster business validation
• Supports agile business development

Conclusion

The Business Model Canvas is one of the most effective frameworks for understanding and designing a business. By breaking a business into nine essential building blocks, teams can quickly identify strengths, weaknesses, opportunities, and risks.

Whether you're launching a startup, improving an existing product, or building an enterprise platform like a QMS Issue Management System, the Business Model Canvas provides a structured way to think about customers, value, operations, and financial sustainability.

A well-maintained Business Model Canvas is not just a planning document—it becomes a living blueprint that evolves alongside your business.

Before deciding whether to build software internally or purchase an existing solution, organizations should first understand which part of the business the software supports. The answer is not purely technical—it is a strategic business decision.

Introduction: Why Build vs Buy Matters

Every organization eventually faces the question:

Should we build this solution ourselves or buy an existing product?

Building software requires significant investment in engineering talent, maintenance, and long-term ownership. Buying software accelerates implementation but may limit flexibility and differentiation.

The right decision depends on how strategically important the capability is to the business.

Understanding Technology Solutions

Not every software system contributes equally to business success.

Some systems create competitive advantage.

Others simply help run the business efficiently.

Understanding this difference is the foundation of Build vs Buy decisions.

Business Domains and Subdomains

A business can be divided into multiple subdomains.

These subdomains generally fall into three categories:

Generic Subdomain

Generic subdomains solve problems that almost every organization has.

Examples include:

• Email
• Payroll
• HR Management
• Accounting
• Calendar
• Authentication

These functions rarely differentiate one company from another.

Recommended Strategy

✅ Buy established products

or

✅ Use SaaS platforms

Examples:

• Microsoft 365
• Google Workspace
• SAP SuccessFactors
• Workday

Building these systems usually offers little business value.

Supporting Subdomain

Supporting subdomains are important but are not the primary reason customers choose your business.

Examples:

• CRM customization
• Vendor Management
• Procurement
• Reporting
• Customer Support

These systems often require organization-specific workflows.

Recommended Strategy

• Buy a mature product
• Customize where necessary
• Outsource implementation if appropriate

This provides flexibility without reinventing existing capabilities.

Core Subdomain

Core subdomains are where the organization creates unique value.

They represent the company's competitive advantage.

Examples:

• Recommendation engine (Netflix)
• Search algorithm (Google)
• Route optimization (Uber)
• Pricing engine
• Risk model
• Trading platform

These capabilities differentiate the business from competitors.

Recommended Strategy

✅ Build internally

Maintain ownership of:

• source code
• business rules
• intellectual property
• innovation

This is where the organization's best engineering team should focus.

Why Categorization Comes Before Build vs Buy

Many organizations make Build vs Buy decisions too early.

Instead, they should first ask:

Which business subdomain does this solution belong to?

Once categorized, the strategy becomes much clearer.

Categorization removes ambiguity and aligns technology investment with business goals.

Core Subdomain: Build or Outsource?

Core subdomains represent the organization's competitive advantage. They contain the business logic, proprietary algorithms, and unique processes that differentiate the company from its competitors.

Unlike generic or supporting subdomains, core capabilities should rarely be purchased as off-the-shelf products because doing so may limit differentiation.

However, organizations don't always need to build everything with a fully in-house engineering team.

The real decision is often:

Build with your own team or Build with a trusted outsourcing partner.

Option 1: Build In-house

Building internally is ideal when:

• The capability is strategically critical.
• Business requirements evolve rapidly.
• Close collaboration with domain experts is required.
• The organization wants complete ownership of the product and intellectual property.
• Long-term innovation is a priority.

Advantages

• Complete control over architecture and roadmap
• Faster business feedback loops
• Strong knowledge retention
• Better protection of intellectual property
• Continuous innovation

Option 2: Build with an Outsourcing Partner

Outsourcing does not mean giving away your competitive advantage.

Instead, it means extending your engineering capacity while retaining ownership of the solution.

The business continues to own:

• Product vision
• Business rules
• Architecture decisions
• Intellectual property
• Source code
• Product roadmap

The outsourcing partner contributes engineering expertise and delivery capacity.

When Outsourcing Makes Sense

• Need to scale engineering teams quickly
• Limited availability of specialized skills
• Faster time-to-market is important
• Temporary increase in development workload
• Cost optimization without compromising ownership

Benefits of Outsourcing Core Development

• Access to experienced engineers
• Faster product development
• Reduced hiring time
• Flexible team scaling
• Lower operational overhead
• Ability for internal teams to focus on product strategy and innovation

The key is that the outsourced team works as an extension of the internal engineering organization, not as an independent owner of the product.

Build vs Outsource Decision

Important Principle

Whether development is performed by internal employees or an outsourcing partner, the organization should retain ownership of the product, architecture, intellectual property, and business knowledge.

The strategic mistake is not outsourcing development—it is outsourcing ownership and decision-making.

Think of outsourcing as adding more skilled engineers to your team, while keeping product strategy, business expertise, and innovation firmly under your control.

             Core Business Capability
                      │
      ┌───────────────┴───────────────┐
      │                               │
 Build In-house                 Build via Outsourcing
      │                               │
      └───────────────┬───────────────┘
                      │
          Business Owns Everything
    • Product Vision
    • Architecture
    • Source Code
    • Intellectual Property
    • Product Roadmap

Build vs Buy Decision Matrix

Decision flow:

New Requirement
        │
        ▼
Identify Business Subdomain
        │
        ├──────── Generic ───────► Buy
        │
        ├──────── Supporting ────► Buy + Customize
        │
        └──────── Core ──────────► Build

Maximizing ROI Through Core Investment

Technology budgets are always limited.

Organizations achieve the highest return on investment by allocating their best resources toward core business capabilities.

Instead of spending months building payroll or email systems, engineering teams should focus on solving problems that directly improve:

• customer experience
• revenue generation
• operational excellence
• competitive differentiation

This creates long-term strategic value.

Common Mistakes

Building Generic Solutions

• Reinvents existing software
• High maintenance cost
• Low business value

Buying Core Capabilities

• Loss of differentiation
• Vendor dependency
• Limited innovation

Treating Every Requirement as "Special"

Not every business process is unique.

Many organizations overestimate the uniqueness of their operations, resulting in unnecessary custom development.

Key Takeaways

• Build vs Buy is a business strategy, not just a technical decision.
• Categorize business domains before selecting a technology approach.
• Buy solutions for generic capabilities.
• Buy and customize supporting capabilities.
• Build core capabilities using the organization's strongest engineering team.
• Invest engineering effort where it creates competitive advantage.
• Businesses maximize ROI by focusing technology investment on their core subdomains.

Final Thought

A successful technology strategy is not about building everything.

It is about building only what makes your business unique and leveraging proven solutions for everything else.

Organizations that consistently apply this principle deliver software faster, reduce costs, and invest where innovation creates the greatest business value.

Sending marketing emails at scale can be one of the most effective ways to reach users, drive engagement, and grow revenue. However, poor practices can quickly lead to spam complaints, high bounce rates, or even domain blacklisting.

Below are essential best practices to ensure your email campaigns are effective, compliant, and well-received.

1. Always Include an Unsubscribe Link

Every marketing email must include a clear and easy way for users to opt out.

Why it matters:

• Required by regulations like GDPR, CAN-SPAM, and similar laws
• Builds trust with recipients
• Reduces spam complaints

Best practices:

• Place the unsubscribe link in the footer
• Make the process one-click or simple (no login required)
• Process unsubscribe requests immediately

2. Avoid Sending Too Many Emails in a Short Time

Email fatigue is real. Overloading users leads to higher unsubscribe rates and lower engagement.

Why it matters:

• Protects sender reputation
• Improves open and click-through rates
• Prevents users from marking emails as spam

Best practices:

• Set a consistent sending schedule (e.g., weekly or bi-weekly)
• Use frequency caps per user
• Segment audiences to control email volume per group

3. Validate Email Addresses Before Sending

Invalid or fake email addresses can damage deliverability and sender reputation.

Why it matters:

• Reduces bounce rates
• Improves inbox placement
• Saves cost on email service providers

Best practices:

• Use real-time email validation during sign-up
• Periodically clean your email list
• Remove hard bounces automatically

4. Segment Your Audience

Not all users should receive the same message.

Why it matters:

• Increases relevance and engagement
• Reduces unsubscribe rates
• Improves conversion rates

Best practices:

• Segment by behavior (clicks, purchases, activity)
• Segment by demographics (location, language, age)
• Segment by lifecycle stage (new users, active users, churned users)

5. Use Double Opt-In for Subscriptions

Double opt-in requires users to confirm their email address before being added to your list.

Why it matters:

• Ensures valid email addresses
• Confirms user intent
• Improves list quality

6. Monitor Deliverability Metrics

Tracking performance helps you identify issues early.

Key metrics to watch:

• Bounce rate
• Open rate
• Click-through rate (CTR)
• Spam complaint rate
• Unsubscribe rate

Best practices:

• Set up domain authentication (SPF, DKIM, DMARC)
• Use a reputable email service provider
• Warm up new domains gradually

7. Write Clear and Honest Subject Lines

Misleading subject lines may increase opens short-term but harm trust long-term.

Best practices:

• Keep subject lines concise
• Avoid spam trigger words (e.g., "FREE!!!", "URGENT!!!")
• Ensure content matches the subject line

8. Optimize Email Content and Design

A well-structured email improves engagement.

Best practices:

• Use responsive design for mobile users
• Keep paragraphs short and scannable
• Include a single primary call-to-action (CTA)
• Avoid overly image-heavy emails (can trigger spam filters)

9. Respect Time Zones and Sending Time

Sending emails at the right time improves engagement rates.

Best practices:

• Analyze user behavior by region
• Schedule emails during active hours
• Test different send times (A/B testing)

10. Maintain a Clean Sender Reputation

Your domain and IP reputation determine whether emails reach inboxes.

Best practices:

• Avoid sudden spikes in email volume
• Regularly remove inactive users
• Monitor blacklists
• Use dedicated IPs if sending at high volume

11. Provide Value in Every Email

Users should feel that opening your email is worth their time.

Best practices:

• Share useful content, not just promotions
• Personalize messages where possible
• Avoid repetitive or irrelevant messaging

Conclusion

Effective bulk email marketing is a balance between compliance, technical setup, and user experience. By following these best practices, you can improve deliverability, maintain trust, and maximize engagement over time.

Useful links and references

Remember: a high-quality email list is more valuable than a large but disengaged one.

1. Domain Registration

Before setting up your mail server, you need a domain.

Steps

1. Register a domain from a registrar:

   • Namecheap
   • GoDaddy
   • Cloudflare Registrar
   • Google Domains (if available in your region)

2. Choose a domain like: example.com

3. Ensure you have access to DNS management.

2. Server Requirements

You need an Ubuntu 24.04 server with:

• Static public IP (important)
• Open ports:
  • 25 (SMTP)
  • 587 (Submission)
  • 465 (SMTPS optional)
  • 143 (IMAP)
  • 993 (IMAPS)
  • 110 (POP3 optional)
  • 995 (POP3S optional)

3. Update System

sudo apt update && sudo apt upgrade -y

4. Install Postfix (SMTP Server)

Installation

sudo apt install postfix -y

During installation:

• Select: Internet Site
• Set system mail name: example.com

Configure Postfix

Edit:

sudo nano /etc/postfix/main.cf

Basic configuration:

myhostname = mail.example.com
mydomain = example.com
myorigin = /etc/mailname
inet_interfaces = all
inet_protocols = all

mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
home_mailbox = Maildir/

smtpd_banner = $myhostname ESMTP

Enable SMTP authentication

Later we integrate with Dovecot:

smtpd_sasl_auth_enable = yes
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_recipient_restrictions =
    permit_sasl_authenticated,
    permit_mynetworks,
    reject_unauth_destination

5. Install Dovecot (IMAP/POP3 Server)

sudo apt install dovecot-core dovecot-imapd dovecot-pop3d -y

Configure Mailbox Format

Edit:

sudo nano /etc/dovecot/conf.d/10-mail.conf

Set:

mail_location = maildir:~/Maildir

Enable IMAP & POP3

Edit:

sudo nano /etc/dovecot/dovecot.conf

Ensure:

protocols = imap pop3

Configure Authentication Socket for Postfix

Edit:

sudo nano /etc/dovecot/conf.d/10-master.conf

Add:

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

6. Install OpenDKIM (Email Signing)

sudo apt install opendkim opendkim-tools -y

Generate DKIM key

sudo opendkim-genkey -s mail -d example.com

Move keys:

sudo mv mail.private /etc/opendkim/keys/example.com/

Configure OpenDKIM

Edit:

sudo nano /etc/opendkim.conf

Add:

Domain                  example.com
KeyFile                 /etc/opendkim/keys/example.com/mail.private
Selector                mail
Socket                  local:/var/spool/postfix/run/opendkim/opendkim.sock

Connect to Postfix

smtpd_milters = local:/var/spool/postfix/run/opendkim/opendkim.sock
non_smtpd_milters = local:/var/spool/postfix/run/opendkim/opendkim.sock

7. DNS Configuration (VERY IMPORTANT)

All DNS records must be configured correctly for deliverability.

7.1 A Record

mail.example.com → x.x.x.x

7.2 MX Record

example.com → mail.example.com (priority 10)

7.3 SPF (TXT record)

example.com TXT

v=spf1 mx ip4:x.x.x.x -all

7.4 DKIM Record

mail._domainkey.example.com TXT

v=DKIM1; k=rsa; p=PUBLIC_KEY_HERE

7.5 DMARC Record

_dmarc.example.com TXT

v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com; adkim=s; aspf=s; pct=100

Recommended later:

p=reject

7.6 MTA-STS (Security)

DNS record:

_mta-sts.example.com TXT

v=STSv1; id=2026070501

Policy file:

Host:

https://mta-sts.example.com/.well-known/mta-sts.txt

Content:

version: STSv1
mode: enforce
mx: mail.example.com
max_age: 604800

7.7 TLS Reporting (TLS-RPT)

_smtp._tls.example.com TXT

v=TLSRPTv1; rua=mailto:tlsrpt@example.com

8. SSL/TLS Setup (Let’s Encrypt)

Install Certbot:

sudo apt install certbot -y

Generate certificate:

sudo certbot certonly --standalone -d mail.example.com

Configure Postfix TLS:

smtpd_tls_cert_file=/etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_security_level=may

Configure Dovecot TLS:

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

9. Open Required Ports

sudo ufw allow 25
sudo ufw allow 587
sudo ufw allow 465
sudo ufw allow 143
sudo ufw allow 993
sudo ufw enable

10. Restart Services

sudo systemctl restart postfix
sudo systemctl restart dovecot
sudo systemctl restart opendkim

11. Testing Email Setup

Check DNS

dig MX example.com
dig TXT example.com

Check DKIM

opendkim-testkey -d example.com -s mail -vvv

Send test email

Use:

• Gmail
• Mail-Tester.com

Check:

• SPF PASS
• DKIM PASS
• DMARC PASS

12. Deliverability Best Practices

To avoid spam filtering:

Always:

• Use verified email lists
• Enable unsubscribe links
• Warm up IP gradually
• Monitor bounce rates

Avoid:

• Bulk sending immediately
• Purchased email lists
• High bounce rates

13. Production Architecture Recommendation

For better scalability:

Apps → Postfix (queue) → DKIM → SMTP → Internet
        ↓
   Dovecot (mailboxes)

For marketing:

• Use separate domain or IP
• Use rate limiting
• Track engagement

14. Summary

A production email server requires:

• Proper DNS (SPF, DKIM, DMARC)
• Correct Postfix + Dovecot configuration
• TLS encryption
• IP reputation management
• Controlled sending behavior

Final Note

Self-hosted email servers give full control but require careful management of:

• reputation
• deliverability
• rate limits

For bulk marketing, consider separating transactional mail and marketing infrastructure to maintain inbox placement quality.

This repository uses a multi-environment GitLab CI/CD pipeline for a Spring Boot Reactive (WebFlux) application.

The pipeline provides:

• Automated build and deployment
• Parallel deployment to multiple test environments
• Mattermost notifications with changelog support
• Automatic handling of unavailable GitLab runners
• Deployment status tracking
• Production deployment authorization controls
• Deployment failure notifications with job logs

Architecture

Test Environments

Both environments build and deploy independently.

Pipeline success requires at least one test deployment to complete successfully.

Pipeline Stages

notify_start
    │
    ▼
build_test
    ├── build_test_th
    ├── build_test_kp
    └── pending_job_monitor_build_test
    │
    ▼
deploy_test
    ├── deploy_test_th
    ├── deploy_test_kp
    └── pending_job_monitor_deploy_test
    │
    ▼
pipeline_complete
    │
    ▼
deploy_prod (manual)

Stage Details

1. notify_start

Purpose

Sends a pipeline start notification to Mattermost.

Features

• Includes pipeline link
• Includes changelog
• Displays recent commits

Example:

🚀 Build STARTED

Changes in this build:

- abc123 Fixed login issue
- def456 Added Kafka consumer
- ghi789 Updated API validation

2. build_test

Jobs

build_test_th

Builds application for TH environment.

build_test_kp

Builds application for KP environment.

Build Process

1. Copy example property files
2. Generate environment-specific property file
3. Run Gradle build

./gradlew clean build \
-Pspring.config.additional-location=file:./application-test.properties

Generated Artifacts

build/libs/spring-boot-template-cqrs-1.0.0.jar
application-test.properties

Artifacts expire after:

1 day

3. Runner Availability Monitoring

pending_job_monitor_build_test

Problem:

If a dedicated runner is unavailable, jobs can remain stuck in Pending.

Solution:

After 30 seconds:

1. Wait configured timeout
2. Query GitLab API
3. Detect pending jobs
4. Cancel stuck jobs automatically

Current timeout:

PENDING_JOB_TIMEOUT=15

4. deploy_test

Jobs

deploy_test_th

Deploys build artifact to TH environment.

deploy_test_kp

Deploys build artifact to KP environment.

Deployment Flow

Step 1

Backup existing JAR

old.jar -> old.jar_bkp

Step 2

Upload new JAR

scp build/libs/app.jar

Step 3

Upload generated property file

scp application.properties

Step 4

Execute remote deployment command

Example:

docker compose restart

or

systemctl restart my-service

Deployment Validation

Deployment only proceeds if:

Build succeeded
AND
Artifact exists

Otherwise deployment is skipped.

5. Runner Availability Monitoring (Deploy)

pending_job_monitor_deploy_test

Automatically cancels deploy jobs that remain pending due to unavailable runners.

6. pipeline_complete

Purpose

Determines final pipeline status.

Success Criteria

Pipeline succeeds if:

TH deployment succeeded
OR
KP deployment succeeded

Failure Criteria

Pipeline fails if:

TH deployment failed
AND
KP deployment failed

Examples:

7. deploy_prod

Type

Manual deployment.

Authorization

Only approved GitLab users can deploy to production.

Validation:

GITLAB_USERS_IDS_FOR_PROD_DEPLOY

contains:

GITLAB_USER_ID

Unauthorized users receive:

❌ User is not authorized for production deployment

Production Deployment Steps

1. Backup current JAR
2. Upload new JAR
3. Upload property file
4. Execute production deployment command
5. Send Mattermost notification

Mattermost Notifications

Start Notification

Color:

Blue

Example:

🚀 Build STARTED

Deployment Success

Color:

Purple

Example:

✅ TH Deploy SUCCESS

Pipeline Success

Color:

Green

Example:

🎉 Pipeline SUCCESS

Failure Notification

Color:

Red

Includes:

• Error details
• Last 50 log lines
• Pipeline link
• Job URL

Example:

❌ Deploy FAILED

Job Log:
...

Changelog Generation

Each pipeline automatically generates a changelog.

Information included:

• Commit hash
• Author
• Commit message

Example:

- abc123 Added Kafka consumer [John]
- def456 Fixed Redis configuration [Mike]

Limits:

Required GitLab Variables

General

APP_NAME

GYRI_TEST_PROPERTY_FILE

GYRI_TEST_MATTERMOST_WEBHOOK
ALL_FAIL_MATTERMOST_WEBHOOK

GITLAB_API_TOKEN

TH Environment

GYRI_TEST_TH_SERVER_USER
GYRI_TEST_TH_SERVER_01
GYRI_TEST_TH_SERVER_01_DEPLOYMENT_PATH

TEST_TH_DEPLOY_COMMAND

TEST_TH_APP_LOG_FILE_PROPERTY
TEST_TH_APP_BASE_SERVER_URL
TEST_TH_SWAGGER_SERVER_URL

TEST_TH_REDIS_HOST
TEST_TH_REDIS_PORT
TEST_TH_REDIS_TOPIC_PREFIX

TEST_TH_MONGO_HOST
TEST_TH_MONGO_PORT
TEST_TH_MONGO_DATABASE
TEST_TH_MONGO_USERNAME
TEST_TH_MONGO_PASSWORD
TEST_TH_MONGO_AUTH_DATABASE

TEST_TH_KAFKA_SECURITY_PROTOCOL
TEST_TH_KAFKA_SASL_MECHANISM
TEST_TH_KAFKA_SASL_JAAS_CONFIG
TEST_TH_KAFKA_SERVERS
TEST_TH_KAFKA_CONSUMER_GROUP_ID
TEST_TH_KAFKA_TOPIC_PREFIX

KP Environment

GYRI_TEST_KP_SERVER_USER
GYRI_TEST_KP_SERVER_01
GYRI_TEST_KP_SERVER_01_DEPLOYMENT_PATH

TEST_KP_DEPLOY_COMMAND

TEST_KP_APP_LOG_FILE_PROPERTY
TEST_KP_APP_BASE_SERVER_URL
TEST_KP_SWAGGER_SERVER_URL

TEST_KP_REDIS_HOST
TEST_KP_REDIS_PORT
TEST_KP_REDIS_TOPIC_PREFIX

TEST_KP_MONGO_HOST
TEST_KP_MONGO_PORT
TEST_KP_MONGO_DATABASE
TEST_KP_MONGO_USERNAME
TEST_KP_MONGO_PASSWORD
TEST_KP_MONGO_AUTH_DATABASE

TEST_KP_KAFKA_SECURITY_PROTOCOL
TEST_KP_KAFKA_SASL_MECHANISM
TEST_KP_KAFKA_SASL_JAAS_CONFIG
TEST_KP_KAFKA_SERVERS
TEST_KP_KAFKA_CONSUMER_GROUP_ID
TEST_KP_KAFKA_TOPIC_PREFIX

Production

GYRI_PROD_SERVER_USER
GYRI_PROD_SERVER_HOST
GYRI_PROD_SERVER_SSH_PORT
GYRI_PROD_SERVER_DEPLOYMENT_PATH

PROD_DEPLOY_COMMAND

GYRI_PROD_MATTERMOST_WEBHOOK

GITLAB_USERS_IDS_FOR_PROD_DEPLOY

GitLab Runner Registration

The pipeline requires three runners:

Install GitLab Runner

Ubuntu/Debian

curl -L \
https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh \
| sudo bash

sudo apt install gitlab-runner

Verify:

gitlab-runner --version

Register TH Runner

sudo gitlab-runner register

Prompts:

GitLab URL:
https://gitlab.example.com

Registration Token:
<project-runner-token>

Description:
th-runner

Tags:
th-runner

Executor:
shell

Register KP Runner

sudo gitlab-runner register

Prompts:

GitLab URL:
https://gitlab.example.com

Registration Token:
<project-runner-token>

Description:
kp-runner

Tags:
kp-runner

Executor:
shell

Register Production Runner

sudo gitlab-runner register

Prompts:

GitLab URL:
https://gitlab.example.com

Registration Token:
<project-runner-token>

Description:
prod-runner

Tags:
prod-runner

Executor:
shell

Verify Registered Runners

sudo gitlab-runner list

Example:

th-runner
kp-runner
prod-runner

Start Runner Service

sudo systemctl enable gitlab-runner
sudo systemctl start gitlab-runner

Check status:

sudo systemctl status gitlab-runner

Deployment Directory Structure

Expected structure on target servers:

template-spring-cqrs-reactjs/

├── server/
│
├── artifacts/
│   ├── application.jar
│   └── application.jar_bkp
│
└── server-config/
    └── application.properties

Failure Handling

The pipeline is designed to tolerate infrastructure failures.

Examples:

TH Runner Offline

TH Build → Failed

KP Build → Success

KP Deploy → Success

Pipeline → SUCCESS

KP Runner Offline

TH Build → Success

TH Deploy → Success

Pipeline → SUCCESS

Both Runners Offline

TH Build → Failed

KP Build → Failed

Pipeline → FAILED

Summary

This pipeline provides:

• Multi-location deployment (TH + KP)
• Spring Boot Reactive build automation
• Automatic changelog generation
• Mattermost notifications
• Job log reporting on failures
• Runner health monitoring
• Automatic cancellation of stuck jobs
• Production deployment authorization
• Artifact backup and rollback preparation
• High availability deployment strategy

If you've recently upgraded to React Redux v9 and are using IntelliJ, VS Code, or another TypeScript-aware IDE, you may have noticed a warning when hovering over connect():

@deprecated
We recommend using the useSelector and useDispatch hooks instead.
See https://react-redux.js.org/api/hooks

If you need to use connect without this visual deprecation warning,
import legacy_connect instead:

import { legacy_connect as connect } from 'react-redux'

This can be surprising, especially because many existing React applications still use connect() extensively.

Let's understand what's happening and explore the available options.

Is connect() Actually Deprecated?

Not exactly.

React Redux v9 adds a TypeScript/JSDoc deprecation annotation to encourage developers to adopt the modern Hooks API:

• useSelector()
• useDispatch()

However, connect() still works and remains supported.

The warning is primarily intended to guide developers toward the recommended pattern for function components.

Example: Existing connect() Implementation

Many applications use a pattern similar to this:

import { useNavigate } from 'react-router-dom';
import { useEffect } from 'react';
import { connect } from 'react-redux';

const ForgotPassword = ({ auth }) => {
  const navigate = useNavigate();

  useEffect(() => {
    if (auth.loggedIn) {
      navigate('/dashboard');
    }
  }, [auth.loggedIn, navigate]);

  return <div>Forgot Password</div>;
};

const mapState = (state) => ({
  auth: state.auth,
});

export default connect(mapState)(ForgotPassword);

This code is completely valid and will continue to work.

The only issue is the IDE warning.

Option 1: Migrate to Hooks (Recommended)

For function components, React Redux recommends using hooks instead of connect().

Before

export default connect(mapState)(ForgotPassword);

After

import { useSelector } from 'react-redux';

const ForgotPassword = () => {
  const auth = useSelector((state) => state.auth);

  return <div>Forgot Password</div>;
};

export default ForgotPassword;

A more complete example:

import { useNavigate } from 'react-router-dom';
import { useEffect } from 'react';
import { useSelector } from 'react-redux';

const ForgotPassword = () => {
  const navigate = useNavigate();

  const auth = useSelector((state) => state.auth);

  useEffect(() => {
    if (auth.loggedIn) {
      navigate('/dashboard');
    }
  }, [auth.loggedIn, navigate]);

  return <div>Forgot Password</div>;
};

export default ForgotPassword;

Benefits

• Less boilerplate
• No mapStateToProps
• Better TypeScript support
• Officially recommended by React Redux

Option 2: Continue Using connect()

If your application already contains many connected components, there is no immediate need to refactor everything.

You can simply keep using:

import { connect } from 'react-redux';

The warning is informational and does not affect runtime behavior.

This approach is often preferred in mature enterprise applications where consistency is more important than adopting the latest syntax.

Option 3: Use legacy_connect

React Redux v9 introduces a special alias called legacy_connect.

Replace:

import { connect } from 'react-redux';

with:

import { legacy_connect as connect } from 'react-redux';

The rest of your code remains unchanged:

const mapState = (state) => ({
  auth: state.auth,
});

export default connect(mapState)(ForgotPassword);

Why Use legacy_connect?

• Removes IDE deprecation warnings
• No behavior changes
• No refactoring required
• Useful for large existing codebases

Recommended Approach for Different Scenarios

New Projects

Use hooks:

useSelector()
useDispatch()

This is the modern React Redux pattern.

Existing Applications

If your project already contains dozens or hundreds of connected components:

import { legacy_connect as connect } from 'react-redux';

can be a practical intermediate solution.

Gradual Migration Strategy

Many teams adopt the following approach:

1. Keep existing connect() components.
2. Use hooks for all new components.
3. Gradually migrate old components when they are modified.
4. Use legacy_connect to suppress warnings during the transition.

This avoids large-scale refactoring while moving toward modern React Redux patterns.

TypeScript Bonus: Typed Hooks

For Redux Toolkit applications, consider creating typed hooks.

// store.ts
export type RootState = ReturnType<typeof store.getState>;
export type AppDispatch = typeof store.dispatch;

// hooks.ts
import { useDispatch, useSelector } from 'react-redux';

export const useAppDispatch =
  useDispatch.withTypes<AppDispatch>();

export const useAppSelector =
  useSelector.withTypes<RootState>();

Usage:

const auth = useAppSelector((state) => state.auth);

This provides stronger type safety and a better developer experience.

Conclusion

The connect() warning in React Redux v9 is not an indication that the API has been removed. It is a recommendation from the React Redux team to use the Hooks API in modern React applications.

You can choose one of three paths:

• Best for new code: useSelector() and useDispatch()
• Best for existing code: continue using connect()
• Best for removing warnings without refactoring: legacy_connect

For most teams, a gradual migration strategy offers the best balance between modernizing the codebase and minimizing risk.

Introduction

After installing Ubuntu, one of the first things many users encounter is the Linux terminal. By default,Ubuntu uses Bash (Bourne Again Shell) as its command-line shell. While Bash is powerful and widely used,many developers and Linux enthusiasts prefer an alternative shell called Zsh (Z Shell).
This guide explains what Zsh is, why it is popular, how to install it on Ubuntu, and what happens during its
first-time setup. The explanations are written with beginners in mind, breaking down every command used
during the installation process.

What is a Shell?

Before understanding Zsh, it is important to understand what a shell is.

A shell is a program that allows users to interact with the operating system through commands. Whenever commands such as the following are executed:

ls
cd
mkdir

the shell interprets those commands and communicates with the operating system to perform the requested actions.

Ubuntu uses Bash (Bourne Again Shell) as its default shell.

What is Zsh?

Zsh stands for Z Shell. It is an advanced shell that provides all the features of Bash while adding many enhancements that improve the command-line experience.

Some of the most popular features of Zsh include:
• Better command auto-completion
• Easier navigation through directories
• Command suggestions and corrections
• Customizable themes
• Plugin support
• Improved productivity for developers

Because of these features, Zsh has become one of the most widely used shells in the Linux community.

Why Use Zsh?

Many users choose Zsh for the following reasons:
Better Auto-Completion
Zsh provides intelligent suggestions when typing commands, file names, and directories.
Improved Productivity
Frequently used commands can be executed more quickly with the help of plugins and shortcuts.
Customization
The terminal appearance can be customized using themes, making it more informative and visually appealing.
Learning Modern Linux Tools
Many developers use Zsh along with frameworks such as Oh My Zsh, making it a valuable tool for anyone learning Linux development environments.

Installing Zsh

Step 1: Updating Package Information
Before installing any software, Ubuntu's package information should be updated.
sudo apt update
Understanding the Command
• sudo = Super User DO. Temporarily grants administrator privileges.
• apt = Advanced Package Tool used to manage software packages.
• update = Downloads the latest package information from Ubuntu repositories.
This command does not upgrade software. It only refreshes the package database so Ubuntu knows which software versions are available.
Step 2: Installing Zsh
Install Zsh using the following command:
sudo apt install zsh -y
Understanding the Command
• sudo = Runs the command with administrator privileges.
• apt = Ubuntu package manager.
• install = Installs a software package.
• zsh = The package to be installed.
• -y = Automatically answers "Yes" to confirmation prompts.
This command downloads and installs Zsh along with any required dependencies.
Step 3: Verifying Installation
After installation, verify that Zsh has been installed correctly.
zsh --version
Understanding the Command
• zsh = Runs the Zsh executable.
• --version = Displays version information.
Example output:
zsh 5.9
Displaying a version number confirms that the installation was successful.
Step 4: Making Zsh the Default Shell
Installing Zsh does not automatically replace Bash. To make Zsh the default shell, execute:
chsh -s $(which zsh)
Understanding the Command
This command consists of multiple parts.
which zsh
• which = Finds the location of an executable program.
• zsh = The program being searched for.
Example output:
/usr/bin/zsh
This shows the location of the Zsh executable.
$(which zsh)
The $() syntax is known as command substitution.
It executes the command inside the brackets and uses its output.
Therefore:
$(which zsh)
becomes:
/usr/bin/zsh
chsh -s
• chsh = Change Shell.
• -s = Specifies which shell should be used.
Therefore:
chsh -s $(which zsh)
effectively becomes:
chsh -s /usr/bin/zsh
This tells Linux to use Zsh as the default login shell for the current user.
After running the command, log out and log back in.

First Launch of Zsh

After logging back in and opening the terminal, Zsh may display a configuration screen called zsh-newuser-install.
This happens because no Zsh configuration files currently exist in the user's home directory. Zsh therefore launches a setup wizard to create an initial configuration.
The setup wizard presents four options.
Option (q)
(q) Quit and do nothing.
• Exits the setup wizard.
• The wizard will appear again the next time Zsh starts.
Option (0)
(0) Exit, creating the file ~/.zshrc containing just a comment.
• Creates a minimal .zshrc file.
• Prevents the setup wizard from appearing again.
• Useful for users who want to configure everything manually.
Option (1)
(1) Continue to the main menu.
• Opens the advanced configuration menu.
• Allows customization of history settings, key bindings, completion settings, and other shell features.
Option (2)
(2) Populate your ~/.zshrc with the configuration recommended by the system
administrator.
• Automatically creates a recommended .zshrc file.
• Applies sensible default settings.
• Recommended for beginners.

Recommended Choice for Beginners
For users who are new to Zsh, selecting:
2
is generally the best option.
This option:
• Creates the configuration file automatically.
• Applies useful default settings.
• Allows immediate use of Zsh without manual configuration.

Understanding the .zshrc File
During the setup process, a file named:
~/.zshrc
is created.
Breaking It Down
• ~ = Home directory of the current user.
• .zshrc = Zsh Run Commands file.
• The dot ( . ) indicates that it is a hidden file.
The .zshrc file stores:
• Shell settings
• Aliases
• Plugins
• Themes
• Environment variables
• Custom terminal configurations
Every time Zsh starts, it reads this file and applies the stored settings.

Verifying the Configuration File
To verify that the configuration file was created successfully:
ls -la ~/.zshrc
Understanding the Command
• ls = Lists files and directories.
• -l = Displays detailed information.
• -a = Shows hidden files.
• ~/.zshrc = Path of the configuration file.
If the file exists, Linux displays its details.

Key Takeaways
Installing Zsh introduces several important Linux concepts:
• Linux supports multiple shells.
• Users can choose their preferred shell.
• Shell behavior is controlled through configuration files.
• Commands can be combined using command substitution.
• Terminal environments can be customized extensively.
Understanding these concepts provides a strong foundation for further Linux learning.

Conclusion
Zsh is a modern and feature-rich shell that enhances the Linux terminal experience through improved auto-completion, customization, and productivity features. Installing Zsh is a simple process, but it also serves as an excellent opportunity to learn about package management, shell configuration, and Linux command-line fundamentals.
For beginners exploring Ubuntu and Linux, Zsh provides a practical introduction to terminal customization while maintaining compatibility with familiar Bash workflows.

Definition:
Kafka, in simple terms, is a distributed-infrastructure used to store, manage and process events in real-time.

In traditional software world, developers visualized data as tables which represents things in real world like for example, items in an inventory, signed-up users, cars in a showroom, etc.
But what about things like item being dispatched from storeroom, user clicking somewhere on your website, driver picking up the client from a particular location?
You guessed it right! These are all events.
So, Kafka helps us process these events in real time, meaning the moment that particular event took place.
Hence, there is no such concept of storing these events in a file or a table for processing them in a batch in future.
However, this does not goes on to conclude that Kafka cannot remember events that already took place. Kafka is built to remember these events.

Lets understand the need for Kafka with the help of this simple example.

Consider a thermostat_readings table.

Now, suppose we need to update the thermostat reading for sensor_id 42 from 24 to 20, we will destructively change the value in temperature column for sensor id 42 from 24 to 20.
So our new table, looks like:

The critical issue here is that, we lost our previous info (context) of the sensor_id in Mumbai, like how quickly the temperature shoots up here or at what time of day it heats up.

To overcome this crucial challenge, Kafka implements logs.

Now you might be wondering what are logs?
Logs can be interpreted as an abstraction used by Kafka to store events as a sequence of data items. Mostly these data items are referred as events and quiet rarely called as messages.
The data items are appended at the very end, and items which are already present are never changed! You never modify these logs.
In Kafka, logs are known as Topics. So topics are where our messages get accumulated. And ALWAYS remember, messages in Kafka are IMMUTABLE! Once you write them into a topic, you cannot modify them. Basically, its an event which has happened and you cannot rewrite history; as same as you cannot change your past, simple as that!

So, in a gist, a topic is analogous to a table in a database, and we have numerous tables in a database; similarly we can have numerous topics in a single Kafka cluster.

NOTE: The messages in a topic can be of different format unlike a schema which needs to be followed in a database table.
The reason behind this is, that Kafka stores these messages as plain bytes. So format or schema doesn't matter.

How do you filter out messages out of a topic then?
Just like we deal with immutable data structures in various programming languages. Create a copy of that topic and filter out the unnecessary messages.

MISCONCEPTION: Topics are Logs. Not queues.
When a queue is read, the item is taken out of the queue and read. Now, nobody else can read that value. In a Kafka topic, when a message is read, it still stays intact. I can comeback after a few years and read that same message again!

More on Kafka message:
Its a value, associated with a key.
For example, consider this JSON:
{
"sensor_id": 42,
"location": "Mumbai",
"temperature": 22,
"read_at": 1700
}
Here, the 'sensor_id' becomes the key. This key is basically an identifier the payload relates to. Generally any unique id is taken as the key. Its not mandatory, but highly recommended to have a key!
We also got the timestamp, which tells us the moment the producer created that message.
The message also has some light-weight headers, and the most important parts of a Kafka message are topic, partition and offset!

Policies available in Kafka to prevent your disk from running out of space:
1. Log retention: delete old data or trim logs by size
2. Log compaction: keep only the latest value per key as sometimes context or history is irrelevant to maintain.

How Kafka scales?
If we limit our Kafka topic to a single node, then we will be restricting the topic's ability to scale to the node's disk space.
Since, Kafka is a distributed system, we partition the topic into several partitions.
Once partitions are created we need to decide to which partition the message goes to.
The message key helps us to make this decision. If the key is NULL, the messages are distributed in a round robin fashion. If not NULL and to ensure that a message with a certain key ALWAYS goes into a particular partition, hash the message key against a hash function mod the number of partitions and the output of this hash function is the partition number where the message goes to. This also helps us order our messages.

A few terminologies:

1. Cluster: A collection of one or more Kafka brokers working together to share the workload and provide redundancy (simply, a network of nodes).
2. Node: The underlying physical or virtual infrastructure that hosts a broker (simply, machine with some disk space for storage purposes).
3. Broker: A single Kafka server process that receives, stores, and fetches data (simply, a Kafka software process running on a node).

Point to note here is that, a broker has access to disk space on that node on which it's running; this disk space is basically SSD which is tightly coupled next to the processor.

Brokers are responsible for handling incoming requests to write new messages to partitions as well as read messages out of them.

Replication (for fault tolerance):

What is Replication factor?
Number of copies created for each partition.

We set the replication factor (n), which in above case is 3. The darker one is termed as Leader (lead replica) whereas the remaining (n-1) copies are called as followers.
Messages are preferred to be written into and read from the leader but in rare scenarios we might write and read from a replica which is nearest to us in the network.
Meanwhile, as messages are written into the lead replica, the followers keep scrapping out the newly written messages from the leader to keep themselves updated and have everything replicated.

Now in case the Broker 1 fails, we still have copies of Partitions 0 (with brokers 2, 3) and 2 (with brokers 3, 4).
Here, Broker 1 had the leader of partition 0. So new leader will be elected for partition 0.

PRODUCER:
A producer is nothing but a Kafka client which writes data into the Kafka partition.
CONSUMER:
A consumer is a Kafka client that reads from the Kafka topic.

In short, anything which is not a broker, is, at the end of the day a producer or a consumer.

REFERENCE:
Apache Kafka 101 - https://developer.confluent.io/courses/apache-kafka/events/

SSH stands for Secure Shell.
It is a secure protocol used to:

• Connect to remote servers
• Clone Git repositories
• Push and pull code securely
• Deploy applications

Instead of using passwords, SSH uses cryptographic key pairs.

Step 2: Understand SSH Key Pair

When you create an SSH key, two files are generated:

How it works:

1. You give your public key to GitLab.
2. GitLab stores it.
3. When you connect, GitLab verifies you using your private key.
4. If matched → Access granted.
   
   No password needed.

Step 3: Why We Use SSH in GitLab

GitLab is used for:

• Hosting repositories
• CI/CD
• Version control
• Collaboration

Without SSH (using HTTPS):

git clone https://gitlab.com/username/project.git
Every push asks for username & password.

With SSH:git clone git@gitlab.com:username/project.git
Push → Done instantly 🚀
That’s why developers prefer SSH.

Step 4: Generate SSH Key on Ubuntu

1️⃣ Open Terminal

Press:Ctrl + Alt + T

2️⃣ Check If SSH Already Exists
ls -al ~/.ssh
If folder doesn’t exist → no problem.

3️⃣ Generate New SSH Key

Recommended modern algorithm:
ssh-keygen -t ed25519 -C "your_email@example.com"
If not supported:
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
Press Enter to accept default location:/home/username/.ssh/id_ed25519
Set passphrase (optional but recommended).

4️⃣ Start SSH Agent
eval "$(ssh-agent -s)"

5️⃣ Add Key to SSH Agent
ssh-add ~/.ssh/id_ed25519

6️⃣ Copy Public Key
cat ~/.ssh/id_ed25519.pub
Copy the entire output.

Step 5: Generate SSH Key on Windows

Option 1: Using Git Bash (Recommended)

1. Install Git for Windows
2. Open Git Bash

Run:ssh-keygen -t ed25519 -C "your_email@example.com"
Keys are stored in:C:\Users\YourUsername\.ssh\

View public key:cat ~/.ssh/id_ed25519.pub

Option 2: Using PowerShell

Open PowerShell and run:

ssh-keygen -t ed25519 -C "your_email@example.com"
Same process..

Step 6: Generate SSH Key on macOS

1️⃣ Open Terminal

Cmd + Space → Terminal

2️⃣ Generate Key

ssh-keygen -t ed25519 -C "your_email@example.com"

3️⃣ Start SSH Agent

eval "$(ssh-agent -s)"

4️⃣ Add Key

ssh-add ~/.ssh/id_ed25519

5️⃣ Copy Public Key

pbcopy < ~/.ssh/id_ed25519.pub

Step 7: Add SSH Key to GitLab

1. Login to GitLab
2. Click Profile → Preferences
3. Click SSH Keys
4. Paste copied public key
5. Click Add Key

Now GitLab trusts your machine.

Step 8: Test SSH Connection

Run: ssh -T git@gitlab.com
If successful, you’ll see:Welcome to GitLab!

Step 9: Clone Using SSH

Instead of HTTPS:
git clone git@gitlab.com:username/project.git
Now: git push
No password required.

Step 10: Security Best Practices

✔ Use ed25519
✔ Protect private key
✔ Add passphrase
✔ Never share private key
✔ Set correct permissions (Linux/macOS):

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519

Bonus: How SSH Works Internally

1. Client connects to GitLab
2. GitLab checks public key
3. GitLab sends encrypted challenge
4. Your private key decrypts it
5. Verified → Access granted

This uses asymmetric cryptography — extremely secure.

Final Conclusion

Setting up SSH keys:

• Improves security
• Removes password dependency
• Enables automation
• Essential for DevOps
• Industry best practice

If you are serious about professional development, SSH is not optional — it’s foundational

This guide outlines the technical requirements, dependency changes, and configuration updates necessary to migrate a Spring Boot application to version 4.0.1.

1. Build Environment Update
Before updating application dependencies, ensure your build tool is compatible with Spring Boot 4.0.1.
Gradle Version: Update to Gradle 9.2.1. This version is required to support the updated Spring Boot plugins and dependencies.

2. Dependency Management
Update your build.gradle.kts dependencies to their compatible versions.
Spring Boot Starters: Ensure all standard starters are updated to the managed versions provided by the Spring Boot 4.0.1 BOM:

• spring-boot-starter-actuator
• spring-boot-starter-data-mongodb-reactive
• spring-boot-starter-security
• spring-boot-starter-webflux
• spring-boot-starter-oauth2-client

Third-Party Library Updates: You must manually update several key libraries to versions compatible with Spring Boot 4:

• Redisson: Update redisson-spring-boot-starter to 4.1.0.
• SpringDoc OpenAPI (Swagger): Update to springdoc-openapi-starter-webflux-ui:3.0.1 to maintain API documentation support.

3. Breaking Change: Jackson 3 Migration
Spring Boot 4.0.1 moves from Jackson 2 to Jackson 3, which introduces significant namespace changes. This will cause compilation errors until references are updated.

Dependency Changes: Replace the standard Kotlin module dependency:

// Remove or Update
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
// Add New Tools Module
implementation("tools.jackson.module:jackson-module-kotlin")
// Maintain backward compatibility if needed for specific libs
implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.20.1")

Code Refactoring Required:

• Imports: Update your import statements for the object mapper.
  From: com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
  To: tools.jackson.module.kotlin.jacksonObjectMapper

• Codecs: If you are using Redisson or custom codecs, update TypedJsonJacksonCodec to TypedJsonJackson3Codec.

4. Configuration Properties Updates
Several properties in application.properties have been renamed or require explicit enablement.
MongoDB Configuration: The spring.data.mongodb prefix has been simplified. You must remove the .data segment from your keys.

Full Mapping:

spring.mongodb.host=192.168.1.122
spring.mongodb.port=27017
spring.mongodb.database=AppDBTest
spring.mongodb.username=admin
spring.mongodb.password=secret
spring.mongodb.authentication-database=admin

Swagger UI: SpringDoc now requires explicit properties to enable the UI and API docs endpoints:

springdoc.api-docs.enabled=true
springdoc.swagger-ui.enabled=true

5. Tooling: Replacing Ktlint with Spotless
Older versions of ktlint may not be supported in this environment. It is recommended to switch to the Spotless Gradle plugin for code formatting.

Implementation: Add the plugin to your build.gradle.kts:

plugins {
    id("com.diffplug.spotless") version "8.1.0"
}

Configuration: Configure Spotless to handle both Kotlin and Java formatting:

spotless {
  kotlin {
    target("**/*.kt")
    ktfmt("0.53")
    trimTrailingWhitespace()
    endWithNewline()
  }

  java {
    target("**/*.java")
    googleJavaFormat("1.19.2")
    removeUnusedImports()
    trimTrailingWhitespace()
    endWithNewline()
  }

  // Bind to compilation tasks
  tasks.named("spotlessCheck") { dependsOn("compileKotlin", "compileJava") }
  tasks.named("spotlessApply") { dependsOn("compileKotlin", "compileJava") }
}

• Update and Install Dependencies

We need the web server and all PHP modules required by Nextcloud.
Run this on Server 1 AND Server 2:

sudo apt update

sudo apt install apache2 php libapache2-mod-php php-mysql php-common php-gd php-curl php-imagick php-mbstring php-xml php-zip php-intl php-bcmath php-gmp unzip -y

• Enable Required Apache Modules

Run this on Server 1 AND Server 2:

sudo a2enmod rewrite headers env dir mime
sudo systemctl restart apache2

• Create the Directory Structure

Run this on Server 1 AND Server 2:

sudo mkdir -p /var/www/html/nextcloud
sudo mkdir -p /var/www/html/nextcloud/data

• Set Permissions

Apache (www-data) must own these folders to write files.
Run this on Server 1 AND Server 2:

sudo chown -R www-data:www-data /var/www/html/nextcloud
sudo chmod -R 755 /var/www/html/nextcloud

• Download and Unzip

Run this on Server 1 AND Server 2:

cd /var/www/html
sudo wget https://download.nextcloud.com/server/releases/latest.zip
sudo unzip latest.zip

unzip will extract everything into the nextcloud folder we created

• Final Permission Fix

Ensure the new files are owned by the web server.

sudo chown -R www-data:www-data /var/www/html/nextcloud

• Configure Apache

Run this on Server 1 AND Server 2 to create the web config:

sudo nano /etc/apache2/sites-available/nextcloud.conf

Paste this content:

<VirtualHost *:80>
    DocumentRoot /var/www/html/nextcloud
    
    <Directory /var/www/html/nextcloud>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
    </Directory>
</VirtualHost>

Enable the site:

sudo a2ensite nextcloud.conf
sudo a2enmod rewrite
sudo systemctl restart apache2

Database setup (Master slave replication)

• Set up the Master (Server 1 Only)

1. Secure the installation:

sudo mysql\_secure\_installation

2. Create the Database:

sudo mysql -u root -p

CREATE DATABASE nextcloud;
CREATE USER 'nc\_admin'@'localhost' IDENTIFIED BY '12345';
GRANT ALL PRIVILEGES ON nextcloud.\* TO 'nc\_admin'@'localhost';

-- Create the User for Server 2 to copy data

CREATE USER 'replicator'@'%' IDENTIFIED BY 'password';
GRANT REPLICATION SLAVE ON \*.\* TO 'replicator'@'%';
FLUSH PRIVILEGES;

3. Enable Binary Logging: Edit /etc/mysql/mariadb.conf.d/50-server.cnf:

bind-address = 0.0.0.0
server-id    = 1
log\_bin      = /var/log/mysql/mysql-bin.log
binlog\_do\_db = nextcloud

Restart & Get Coordinates:

sudo systemctl restart mysql
sudo mysql -u root -p -e "SHOW MASTER STATUS;"

• Set up the Slave (Server 2 Only)

1. Do NOT create a database. (It will copy Server 1's DB).

2. Configure Config File: Edit /etc/mysql/mariadb.conf.d/50-server.cnf:

bind-address = 0.0.0.0
server-id    = 2
log\_bin      = /var/log/mysql/mysql-bin.log
binlog\_do\_db = nextcloud

3. Restart:

sudo systemctl restart mysql

• Connect Slave to Master

1. Run on Server 2: Note: If servers are on different networks (192.168 vs 10.10), open the SSH tunnel first.

STOP SLAVE;
CHANGE MASTER TO
MASTER\_HOST='192.168.0.135', -- Or '127.0.0.1' if using Tunnel
MASTER\_USER='replicator',
MASTER\_PASSWORD='password',
MASTER\_LOG\_FILE='mysql-bin.000001', -- From Phase 2, Step 1
MASTER\_LOG\_POS=123;                 -- From Phase 2, Step 1
START SLAVE;

2. Copy Config to Server 2:

On Server 1: Open /var/www/html/nextcloud/config/config.php and copy the text.

sudo nano /var/www/html/nextcloud/config/config.php

On Server 2: Create the file:
Paste the text.

sudo nano /var/www/html/nextcloud/config/config.php

Important Change: Find the line 'dbhost' => 'localhost', and ensure it stays localhost (so Server 2 reads its own replica DB).

File Synchronization (Unison)

• Install Unison (Both Servers)

sudo apt install unison -y

• Configure SSH Key (Server 2 -> Server 1)

ssh-keygen -t ed25519
ssh-copy-id gyriexp@192.168.0.135

• Create Sync Profile (Server 2)
  Create ~/.unison/default.prf:

root = /var/www/html/nextcloud/data
root = ssh://gyriexp@192.168.0.135//var/www/html/nextcloud/data
auto = true
batch = true
prefer = newer
perms = 0     # Ignore permissions

• Running the Sync

Run this sequence whenever you want to backup files:

1. On Server 1 (Unlock):

sudo chown -R gyriexp:gyriexp /var/www/html/nextcloud/data

2. On Server 2 (Sync):

unison default

3. On Server 1 (Lock):

sudo chown -R www-data:www-data /var/www/html/nextcloud/data

Special conditions

Server 1 Fails → Make Server 2 Master → Restore Server 1 as Slave

• Promote Server 2

Run these on Server 2 to make it the independent Master.

1.Stop it from looking for the dead master:

sudo mysql -u root -p -e "STOP SLAVE; RESET SLAVE ALL;"

2.Enable Writes (if Read-Only was on)

sudo mysql -u root -p -e "SET GLOBAL read\_only = OFF;"

3. Update Nextcloud Config: Ensure config.php points to localhost.

sudo nano /var/www/html/nextcloud/config/config.php

Ensure: 'dbhost' => 'localhost',

• Server 1 is Back (Make it the Slave)

1. Create Backup on Server 2:

Run on Server 2

sudo mysqldump -u root -p --all-databases --master-data > production\_backup.sql

2. Send Backup to Server 1:

Run on Server 2

scp production\_backup.sql user@server1\_ip:/home/user/

3. Import & Configure Slave on Server 1:

Run on Server 1

sudo mysql -u root -p < /home/user/production\_backup.sql

4. Start Replication on Server 1:

Run on server 2

sudo mysql -u root -p -e "SHOW MASTER STATUS;"

Run on server 1

sudo mysql -u root -p
STOP SLAVE;
CHANGE MASTER TO
MASTER\_HOST='Server2\_IP',
MASTER\_USER='replicator',
MASTER\_PASSWORD='password',
MASTER\_LOG\_FILE='mysql-bin.00000X',  -- Value from Server 2
MASTER\_LOG\_POS=1234;                 -- Value from Server 2
START SLAVE;

• Correct Unison File Sync

Since Unison is bidirectional, you usually don't need to change the command if the config is on Server 2. It will detect Server 2 has "New files" and Server 1 is empty/old, and it will push them automatically.

\# Unlock Server 1
ssh user@server1\_ip "sudo chown -R user:user /var/www/html/nextcloud/data"

\# Run Sync (Pushes S2 data to S1)
unison default

\# Lock Server 1
ssh user@server1\_ip "sudo chown -R www-data:www-data /var/www/html/nextcloud/data"

Normal chain is S1 → S2 → S3. Server 2 dies. You need S1 to sync directly to S3. When S2 returns, S3 updates S2.

• Bypass Server 2 (S2 is Down)

1. Create a "Bypass Profile" on Server 3:

nano ~/.unison/bypass.prf

Paste this:

\# Sync Local (S3) with Server 1 (Skipping S2)
root = /var/www/html/nextcloud/data
root = ssh://user@server1\_ip//var/www/html/nextcloud/data
auto = true
batch = true
prefer = newer
perms = 0

2. Run the Bypass Sync: Run this on Server 3 whenever S2 is down.

\# 1. Unlock Server 1
ssh user@server1\_ip "sudo chown -R user:user /var/www/html/nextcloud/data"

\# 2. Sync S3 <-> S1
unison bypass

\# 3. Lock Server 1
ssh user@server1\_ip "sudo chown -R www-data:www-data /var/www/html/nextcloud/data"

• Server 2 is Back (Restore & Fill Gaps)

1. Sync S3 → S2 (Run on Server 3): We create a temporary profile to push data back to the recovered Server 2.

nano ~/.unison/restore\_s2.prf

\# Sync Local (S3) to Server 2
root = /var/www/html/nextcloud/data
root = ssh://user@server2\_ip//var/www/html/nextcloud/data
auto = true
batch = true
prefer = newer
perms = 0

2. Run the Restore Sync:

\# 1. Unlock Server 2 (The recovered server)
ssh user@server2\_ip "sudo chown -R user:user /var/www/html/nextcloud/data"

\# 2. Sync S3 -> S2
unison restore\_s2

\# 3. Lock Server 2
ssh user@server2\_ip "sudo chown -R www-data:www-data /var/www/html/nextcloud/data"

3. Resume Normal Operations: Now that S2 is up-to-date, you can go back to your original setup (S1 syncing to S2, and S2 syncing to S3).

• step 1: Verify Server 2 is "Clean"

Run on Server 2:

ls -l /var/www/html/nextcloud/data/

• Step 2: Resume Server 1 → Server 2 Sync

1. Unlock Server 2
   Run on Server 1

ssh user@server2_ip "sudo chown -R user:user /var/www/html/nextcloud/data"

2. ssh user@server2_ip "sudo chown -R user:user /var/www/html/nextcloud/data"

unison default

3. Lock Server 2:

ssh user@server2_ip "sudo chown -R www-data:www-data /var/www/html/nextcloud/data"

• Step 3: Resume Server 2 → Server 3 Sync

Now that S2 is updated by S1, it passes those updates down to S3.
Run on Server 2: (Assuming you have a profile named sync_to_s3 that points to S3).

Unlock Server 3:

ssh user@server3_ip "sudo chown -R user:user /var/www/html/nextcloud/data"

Run the Standard Sync:

unison sync_to_s3

Lock Server 3:

ssh user@server3_ip "sudo chown -R www-data:www-data /var/www/html/nextcloud/data"

Modern authentication systems are moving beyond passwords. With the rise of passwordless authentication, One-Time Passwords (OTPs) delivered over email or SMS are becoming a trusted standard for secure and seamless user logins.

However, implementing an OTP-based login mechanism that is scalable, reactive, and audit-compliant requires more than a simple verification service. It demands an architecture that supports asynchronous workflows, event traceability, and reactive scalability — all while keeping the system maintainable and extendable.

In this post, we’ll walk through the conceptual design of a Reactive OTP Login System built using Spring Boot, CQRS (Command Query Responsibility Segregation), and Event Sourcing.
We’ll focus purely on the architecture and system flow — without diving into code — to illustrate how such a system achieves high throughput, clean separation of concerns, and full auditability.

Architectural Overview

At its core, this system adopts a Reactive CQRS + Event Sourcing design:

• Commands modify the system state (e.g., logging in a user).
• Queries fetch or compute data without altering state (e.g., generating OTP for a user).
• Events record every state transition as an immutable fact.
• Reactive Streams (Mono/Flux) ensure non-blocking, backpressure-aware execution.

Step 1: Requesting an Email OTP

The OTP generation process begins when a user initiates a login using their email.

Conceptual Flow

1. The client sends a request to /api/v1/user/request/emailloginotp/{email}.
2. The Query Controller constructs a RequestEmailLoginOTPQuery, encapsulating the user’s email and tenant metadata.
3. The Query Dispatcher routes the query to the appropriate Query Handler.
4. The Query Handler verifies whether the user exists and is active using a reactive repository.
5. If valid, it invokes EmailLoginOTPService to generate a unique OTP.
6. It then emits an EmailOTPSentEvent, which is published to Kafka for asynchronous notification delivery and audit tracking.
7. The API responds with a confirmation that the OTP request was processed successfully.

Step 2: Validating the Email OTP (Login Process)

Once the user receives the OTP, they can authenticate using it.

Conceptual Flow

1. The client submits a POST request to /api/v1/user/validate/emailloginotp with email and otp.
2. The Command Controller validates the OTP through the EmailLoginOTPService.
3. On success, it dispatches a LogInWithEmailOTPCommand to the Command Handler.
4. The Command Handler performs several key actions:
   • Rehydrates the user’s aggregate state from the Event Store.
   • Raises a LoggedInWithEmailOTPEvent to represent successful login.
   • Persists this event back to the Event Store.
5. The Event Handler listens for the event, updating the read projection (user table) to reflect login status.
6. The JWT Token Provider issues a new access token for the authenticated session.
7. The client receives a success response along with the token.

Event Sourcing Lifecycle

Event Sourcing lies at the heart of this system.
Every change in user state — from OTP sent to login success — is captured as a domain event and stored immutably in the Event Store.

Conceptual Event Chain

Replaying these events allows the system to rebuild state from history, ensuring full traceability and fault recovery without traditional database state dependencies.

Security and Compliance Considerations

Security is baked into every stage of the pipeline:

• OTP expiration, uniqueness, and rate limiting are handled by the EmailLoginOTPService.
• JWT tokens provide stateless, secure session management.
• Audit events ensure all critical actions are logged for compliance.
• Tenant-aware scoping enforces strict data isolation in multi-tenant setups.

By combining event logs with JWT-based access, the system achieves both security and observability.

Conclusion

Building an OTP-based authentication system using Reactive, CQRS, and Event Sourcing principles enables:

Scalable, non-blocking performance under heavy traffic
Strict auditability with event logs
Fault tolerance through event replay
Extensibility for future login channels or audit pipelines
Seamless user experience with real-time responsiveness

In essence, this architecture transforms a simple OTP mechanism into a production-grade authentication framework — one that is reactive by nature, event-driven by design, and compliant by default.

Enterprise applications that export Excel reports with embedded documents face a critical security challenge. The straightforward implementation exposes internal system identifiers:

https://app.example.com/api/document/12345/view?tenantId=67890

This reveals:

• Internal document IDs (12345)
• Tenant identifiers (67890)
• API structure and endpoints

Once Excel files are distributed, these identifiers persist indefinitely, creating ongoing security exposure even if the URLs require authentication.

The challenge intensifies when requirements include:

• Embedding images directly in Excel cells
• Providing download links for full-resolution access
• Supporting non-image documents (PDFs, DOCX) as clickable links

Solution: Short URL Redirection

Replace direct URLs with temporary, opaque redirect tokens:

Before:

https://app.example.com/api/document/12345/view?tenantId=67890

After:

https://app.example.com/api/shorturl/7k9x-m2p4

The short URL provides:

• Complete opacity (no exposed identifiers)
• Automatic expiration (configurable timeframe)
• Full auditability (access tracking)
• Universal compatibility (images and documents)

Architecture

Phase 1: Report Generation

Token Generation:

For each document in the export:

1. Generate cryptographically secure random token (e.g., 7k9x-m2p4)
2. Store mapping in database:

short_code:    7k9x-m2p4
original_url:  /api/document/12345/view?tenantId=67890
expires_at:    2025-11-06 14:30:00 UTC
created_by:    user@example.com

Content Handling:

Images:

• Fetch image via secure internal URL (server-side)
• Embed binary data directly into Excel cell
• Attach short URL as clickable hyperlink
• Result: Thumbnail preview with download link

Non-Images:

• Insert short URL as clickable hyperlink only
• Result: Text link for download/viewing

Excel Generation:

Use server-side libraries (Apache POI, OpenPyXL, EPPlus) to:

• Embed image binary data in cells
• Configure hyperlinks to short URLs
• Ensure zero internal identifiers in output

Phase 2: Document Access

User clicks link in Excel:

1. Request: Browser opens GET /api/shorturl/7k9x-m2p4

2. Validation: Public endpoint checks:

   • Token exists in database
   • Token not expired
   • Log access attempt

3. Redirect: Return HTTP 302:

   Location: https://app.example.com/api/document/12345/view?tenantId=67890
   

4. Authentication: Browser follows redirect with existing session, normal authentication applies

User experience: Click → Document opens

System process: Validate → Redirect → Authenticate → Deliver

Technical Implementation

Database Schema

Security Configuration

Redirect Endpoint:

• Path: /api/shorturl/**
• Authentication: None (public access)
• Rate limiting: Required (prevents enumeration)
• Validation: Token existence and expiration only

Token Specification:

• Length: Minimum 12 alphanumeric characters
• Entropy: Cryptographically secure random generation
• Character set: URL-safe (a-z, A-Z, 0-9)
• Collision handling: Database unique constraint with retry

Expiration Policy

Security Benefits

1. Information Protection

• Internal IDs completely hidden
• API structure remains confidential
• Zero architectural details exposed

2. Temporal Controls

• Automatic expiration reduces long-term risk
• Distributed files lose access over time
• No indefinite credential exposure

3. Audit Capability

• Track every access attempt
• Record timestamp, IP, user identity
• Enable anomaly detection and compliance reporting

4. Access Management

• Revoke tokens immediately if needed
• Extend expiration programmatically
• Tenant-level and user-level controls

Performance Considerations

Token Generation:

• Batch create during report generation
• ~1ms per token generation
• Minimal database overhead

Lookup Performance:

• O(1) with proper indexing
• Cache active tokens (optional)
• Typical response time: <10ms

Cleanup:

• Schedule periodic deletion of expired tokens
• Run during off-peak hours
• Archive audit data before deletion

Image Processing:

• Server-side fetch and embed: ~50-100ms per image
• Memory efficient with streaming
• No browser limitations

Example Workflow

Scenario: Report with 30 images, 20 PDFs

Generation:

1. Generate 50 short URLs: ~50ms
2. Fetch and embed 30 images: ~2-3 seconds
3. Build Excel file: ~500ms
4. Total: ~3-4 seconds

Usage:

• 5 users, 7-day period
• 150 total accesses across 50 documents
• Complete audit trail maintained
• Zero ID exposure
• Automatic expiration after policy period

Implementation Checklist

Required Components:

• Short URL mapping database table with proper indexes
• Token generation service (cryptographically secure)
• Public redirect endpoint (unauthenticated)
• Token validation logic (existence + expiration)
• Audit logging mechanism
• Scheduled cleanup job for expired tokens

Security Requirements:

• Rate limiting on redirect endpoint
• Cryptographic random token generation
• Proper expiration policy configuration
• HTTP 302 redirect (not 301 permanent)
• Error handling (404 for invalid/expired tokens)

Server-Side Excel Library:

• Java: Apache POI
• Python: OpenPyXL
• .NET: EPPlus

Why Server-Side:
Browser-based libraries (xlsx-js-style, etc.) cannot embed images in Excel cells. Server-side libraries provide full binary embedding capability.

Conclusion

The short URL redirection pattern solves document embedding security through architectural indirection. By replacing direct URLs with temporary tokens, systems achieve:

• Complete identifier masking (no IDs exposed)
• Automatic expiration (temporal security)
• Full auditability (compliance ready)
• Seamless UX (transparent to users)

This approach is applicable beyond Excel exports—anywhere temporary, auditable access to secure resources is required without exposing internal architecture.

Key Takeaway: Never expose internal identifiers in exported files. Use opaque, expiring tokens as an indirection layer between distributed content and secure resources.

Technical Stack Considerations:

This pattern integrates with existing authentication infrastructure. The redirect endpoint bypasses authentication (public), while the target URL enforces normal security. Token validation provides the security boundary, relying on cryptographic randomness and expiration rather than session state.

Broader Applications: PDF reports, email notifications, third-party integrations, mobile apps, automated distribution systems—any context requiring temporary resource access without persistent authentication.

• mongod - start mongo db server instance
• mongo - start mongo db command prompt to execute commands
• show dbs - show databases
• mongoimport -d [DATABASE_NAME] -c [COLLECTIONS] --file [FILE_NAME] - import into database and collection from file
• mongoexport -d [DATABASE_NEM] -c [COLLECTIONS] --out [FILE_NAME] - export db in json
• mongodump - binary export
• mongorestore - restore binary export
• bsondump
• mongostat
• use [DB_NAME]- change/switch DB. There is NO command to create DB
• db - set variable as DB (after user [DB_NAME])
• db.[COLLECTIONS] - e.g. db.links. Don't have to create collections (similar to DB)
• db.[COLLECTIONS].count() - to check number of documents. e.g. db.links.count()
• db.[COLLECTIONS].insert({[JSON_DATA]}) - e.g. db.links.insert({title: "google search", url:"www.google.com", comment:"top search engine", tags:["default","main page"], saved_on: new Date()})
• db.[COLLECTIONS].save({[JSON_DATA]}) - e.g. db.links.save({title: "yahoo search", url: "www.yahoo.com", saved_on: new Date()})
• db.[COLLECTIONS].find() - e.g. db.links.find(). Query the database to find the documents

Examples

• show collections - show collections for db
• db.users.insert({"name":"username"}) - insert collection user with data
• db.users.count() - show count of collections
• db.users.find() - find all data from collection
• db.users.find().explain() - explain execution planner for query to find data
• db.users.find().explain("executionStats") - more information on explain
• db.users.createIndex({number:1}) - to create an index on collection
• db.users.update({"name":"username"}, $set: {"first_name":"Name"})" - $set update record
• db.users.update({"name":"username"}, {$addToSet: {"hobbies":"running"}})" - $addToSet update array with new value
• db.users.remove({"name":"username"}) - remove record
• db.users.drop() - drop entire users collection

Notes

• bson: binary JSON
• Collection: Like an RDBMS table; but collection has no schemas.
• Document: Like an RDBMS record or row. There are bson documents.
• Field: Like an RDBMS column; {key: value}