Background

World Perspectives publishes agricultural analytical reports, and sells them to primarily large government and corporate customers. They have been doing so in print and downloadable PDF format for over 30 years. To meet the demands of modern readers and launch themselves into the 21st century, the company required a custom online solution that would allow their clientele to subscribe, read, and receive relevant notifications about articles - all online.

Technology

The system we designed and built is based on a backbone of PHP and MySQL, with Laravel as our wonderful framework. Secure, recurring subscription payments are handled using Stripe, and Elasticsearch runs as a service in a JVM to provide intelligent search capabilities. NPM, Composer, and Bower provide dependency management. A Grunt stack is used to compile Sass to CSS, as well as concatenate and minify CSS and JavaScript to optimize load time.

Interesting Challenges

Existing data

The client already had an existing database of 20,000 or so articles along with current user accounts and subscriptions - all of which of course needed to be retained. In order to ensure that all of the legacy data was retained in the new system, we wrote several migration scripts that automated the process of converting old data into new data. Whenever you're traversing data from an external source like that, especially at quantities large enough that manual proof-reading is not an option, special care must be taken to ensure accuracy. And that's exactly what we did!

PDF content and searchability

Some of the articles exist entirely in PDF format. Since all content needed to be searchable, and since we were especially concerned with delivering relevant data to users based on that content, we needed to make PDFs searchable. The solution was to parse the PDFs using spatie/pdf-to-text, extract the content, and index it along with the rest of the content in Elasticsearch. Every time an article is imported, created, or edited the PDF content is parsed and updated in the Elasticsearch index.

Elasticsearch status

Search is critical to user experience for this application. In the event that the Elasticsearch JVM crashes on the server, it needs to be restarted automatically. In order to ensure that search is working, we run a console command every minute which checks the Elasticsearch service status. In the event that it is not running, the automated command attempts a restart. As a last resort, if the service cannot be automatically restarted, the development team will be notified. This fail-safe scenario has not happened since going live :)

Mailgun and mailing list members' personalized content

Mailgun allows us to create a mailing list of opt-in members, create templates which contains placeholders for variables, and then programmatically trigger templates to send out to all of the mailing list members, with some values passed for the variables. These variable values are applied to all emails sent out to the entire mailing list. But... what if it is critical that each individual recipient gets personalized, unique content? One of the key ingredients in the revamping of the World Perspectives service was targeting end users with relevant content. We spent considerable time tweaking algorithms to ensure that each user receives the most recent, most interesting content specific to them. Because of the number of mailing list members it was not feasible to hit the mailgun API over and over, once for each member - so how did we do it? In order to ensure that each recipient receives their own unique content, we run a series of scheduled tasks that prepare personalized content for each user, store that content on the mailgun record for each individual mailing list member, and then finally deliver newsletters to all members in one fell swoop.