When starting my work in the e-commerce sector back in 1999, the basic concept - from a merchant’s process point of view – was pretty simple and always the same: Push product information into a database and show it in an online catalog. Making products shoppable was the main objective in the early 2000s. The internet was slow and the range of internet-ready devices limited to a desktop computer. Some three-column designs filled with some basic product information plus a small (bad quality) image were already enough to surpass buyers’ expectations – and enabled merchants of every size to sell products online. The commerce platforms we used at that time allowed merchants to add more product information, such as stock availability, discounts, high-res images, filters, quick purchase options, reviews.
But this evolution came at a cost: A cost of complexity and the need for more computing power to keep the buyers’ shopping experience great. Actually, we didn’t use that term ‘experience’ then - but we knew that browsing in a slow online shop just isn’t fun. So we started finding ways to make response times faster.
Using caching technology to improve performance
It was in 2008 when we got challenged by a big aviation enterprise with over 20,000 employees that wanted to buy merchandise in a Magento store at the same time. I had to get creative in order to make that possible, considering the fact that we could only deliver 8-12 pages per second with the servers we had back then. After three days of frustration, I came up with the idea of a simple full-page caching: It checked if a user’s request already had a cookie - and if not, we saved the page Magento returned and reused it for the following visitors. After fine-tuning that basic idea, the very same servers were able to deliver up to 6,000 pages per second. That was mind blowing for us and as a result, our system never suffered of bad performance afterwards.
The next step was a project for a huge telecommunication company in 2011, when we first got in touch with Varnish – a reverse proxy that was able to save a server response based on a fully adjustable ruleset. It improved our simple caching idea from 2008 by enriching it with user-specific dynamic page elements, which were injected before the page was returned to the buyer’s device.
This was a huge step forward, because we were now able to deliver all shop pages – including those for logged in users and/or users who already have added something to their shopping cart or wish list – at a speed of 10-15,000 pages per second.
Managing product catalogs with search technology
These caching technologies worked well for us for quite some time – but never solved the root cause of our problem: Our full-featured e-commerce platforms still needed a lot of time for initial page loads! And as e-commerce sites do not only consist of (almost) static product pages but also category pages with filter and sorting options, the full-page cache approach always suffered from cache misses and potential buyers were faced again with slow page load times.
In 2012, we worked for a global brand selling exclusive wines online. Products were assigned to multiple categories (wine type, winemaker, region etc.) – each with a specific extensive set of filter options. This is when a colleague came up with the idea to use Lucene search technology which allowed filtering in a huge collection of documents (or products) blazingly fast.
Search servers are indexing at the moment information is stored in their database. And since filtering for a product attribute like ‘manufacturer’ is simply a search which is executed against the search server’s index, product information is found much faster than in our shop’s database. However, most of the search implementations we’ve seen so far do not save all the product information in the search server. Instead, they only save search or filter relevant information in it and then use the products’ IDs/SKUs returned by the search server to lookup the product information in the shop’s database. I’m not sure why this is the case - maybe a lack of trust? But obviously, a search server can save all product information necessary while allowing much faster access to that information at the same time.
We started to build our own search module based on Apache Solr and switched quickly to Elasticsearch after it became stable. First we needed an indexer to copy all product information from the shop’s MySQL database to the search server. Afterwards we enabled the Magento frontend to look up and retrieve all product information used for page generation from the search server. This way, we were able to eliminate all product catalog related lookups to the shop’s database.
This little technological exercise brought us down to first hit response times below one second for all kinds of filter and search combinations! In addition, it enabled our merchants to create dynamic landing pages and to manage search results by leveraging capabilities like boosting on the search server.
Leveraging the PWA advantage
Progressive Web Apps embrace many aspects of modern web development in a mobile first world. For us as a technology company helping merchants to deliver great user experiences and to keep up with constantly increasing user expectations, there’s that one winning reason for PWA: the ability to cache and render product information on the device it is viewed on.
The classic approach of e-commerce applications is to render the HTML on the server and then deliver it to the user. This requires a lot of computing power and therefor a lot of expensive hardware. By using the computing power of your users’ devices to render the shop’s pages you get out of the auto-scale server hell and actually save big money each month. So this is a win-win-win situation: The merchant saves money, the user gets a great shopping experience and the developers are finally able to deliver effortlessly scalable e-commerce solutions.
How is that possible? Great PWA solutions like VueStorefront simply provide a very lightweight interface to the Elasticsearch server where all the catalog and CMS information is stored. It takes only 5-10 milliseconds to retrieve the product information from Elasticsearch and another 20-30 milliseconds to send this information to the user’s device. Browsing through a product catalog is now possible without any noticeable delays - it’s hard to believe that these responses came from a server. But in fact it is the result of a smart technological evolution that brings catalog information to the edge of your data center and uses the device’s computing capacity to render the shop’s pages. This all works without full-page caching and all the pain points that technology incorporates.
Why investing in a PWA future?
Ten years ago, we invested a lot of time in building caching layers making our dynamic e-commerce pages static – in an effort to deliver fast page response times. About seven years ago, we invested thousands of development hours to allow individual browsing at an acceptable speed. And in today’s mobile world, we are convinced that individual shopping experiences based on PWA will be more successful than anything else. And being a first adopter of the PWA technology means being ahead of the curve and having an advantage over your competitors adopting this technology in 2-3 years maybe.
If you want to know, why we are so thrilled about this technology, just grab your mobile phone and browse through the VueStorefront demo store. And after a few clicks, just turn on airplane mode – and continue shopping!
As a Magento Enterprise Solution Partner for more than ten years, we are proud to be recognized as a Core Partner for VueStorefront contributing to this great technology and elevating the commerce experience and our merchants to the next level.
