Eager Loading is Bad, Bad…

Introduction

In web applications, data management plays a crucial role in providing a seamless user experience. When dealing with a complex system, efficient data retrieval becomes even more essential. However, using eager loading as a solution to data retrieval can have its drawbacks. In this article, we’ll explore the challenges of eager loading and discuss alternative approaches to manage data more efficiently.

image by Mahsa Monshizadeh, #WomanLifeFreedom

Why Eager Loading Is Bad

Eager loading is a technique where an application retrieves all related data from the database at once, often resulting in a massive amount of data being transferred between the server and the client. While this approach may simplify front-end tasks, it can create several issues:

1. Performance: Eager loading burdens the database, as it has to process complex queries and retrieve vast amounts of data in a single request. This can result in slow performance and increased response times.
2. Bandwidth: Transferring large amounts of data can consume significant bandwidth, leading to slow loading times and a poor user experience, especially on slower internet connections.

Consider a loan and insurance affiliate application that allows individuals or couples to apply for loans. In this scenario, you may have a LoanApplication an entity connected to one or two Customer entities, as well as OwnerAgent, CreatorAgent, AcceptedByAgent, SmsMessages, EmailMessages, and multiple LoanOffers from various providers.

To display a list of loan applications in the front end, you might want to include the main customer and co-customer names, the number of SMS and email messages, and the creator, owner, and accepted by agent information. If you decide to use eager loading to fetch this data, you could potentially transfer several megabytes of data for just 500 applications, leading to performance issues.

applications = LoanApplication.query.options(
    db.joinedload(LoanApplication.customers),
    db.joinedload(LoanApplication.sms_messages),
    db.joinedload(LoanApplication.email_messages),
    db.joinedload(LoanApplication.creator_agent),
    db.joinedload(LoanApplication.owner_agent),
    db.joinedload(LoanApplication.accepted_by_agent),
).all()