Nobody knows what content strategists really do, apparently even me.
On paper, I’m a digital/content strategist typically hired to redesign and implement new websites. I’ve done that job 3 times, and embarrassingly none of those times involved a search team. Why are you reading this (and why am I writing this)? Content- and digital-strategy focused people who touch the internet need to know more about search.
Whether you’re a transformation specialist, content strategist, designer (UX or graphic), solutions architect or even a journalist, you need to understand SEO and SXO, grasp how to deliver the tangible results users are expecting and stay abreast of what’s going on in the AI space. Keep your eyes peeled for emerging technologies such as voice and visual search. Build your work around these blossoming developments or, at the very least, set your organization up for success by structuring your content and staying vigilant about good metadata on your content assets.
The most important thing about your web content is not how pretty it is or even what your traffic numbers look like. It’s making sure that machines can find stuff when humans ask them to look for it. Having a well-executed, thought-out search experience on your site once humans get there is your call—and remember it takes a lot more work than you realize.
As of 2017, only 30% of web visitors were using on-site search. Last month at a marketing automation conference, one speaker told me that only 10% of users bother with on-site search tools on nonprofit websites at best—most numbers are closer to 2%. Today’s gold standard for search experience is Google. Nobody can touch them, especially not a nonprofit with a handful of folks on its web or IT teams and nary a data scientist or engineer in sight.
I hail from the content management side of things, having come up in print media, then pivoting to digital in 2007. I don’t have an engineering degree; I majored in English. And the longer I do digital stuff, the more I realize I need to know more about engineering. So, in that spirit I dedicated almost all of 2019 to attending conferences and learning more about search, programming and machine learning.
Now that we’re clear that I’m a search noob, let’s go a little easy on the fact that I may say some things (or cite some products, or describe some attributes) in a less-than-perfect way in this article.
The Non-Search Project: Wins, Oversights and Mistakes
In 2018, I oversaw the redesign and deployment of a nonprofit website that attracts more than 1 million visitors a year. During the project, my team came face-to-face with what turned out to be the biggest technical obstacle of our careers: on-site search.
The ASM website’s information architecture was a bit ahead of the nonprofit curve. We also streamlined 20k pages down to a tidy 900 and dug deep as an organization to think about our audiences, revenue streams and user needs. In hindsight, however, the biggest oversight of this project was not applying that same level of thoughtfulness to our on-site search tool.
Full transparency: Our website was on an accelerated timeline for a launch date of December 2018, and our search deployed after being up and running for about, oh, 48 hours. While we did have validated user stories and design thinking around search pages, serious search design only began about 2 months prior to launch.
We got quite a few key components right while building our site, which can lay a great foundation for on-site search, including:
- structured content (we actually built out content models that our developers mimicked in the CMS)
- a taxonomy/controlled vocabulary that served as the foundation for our tagging structure
- faceted search
- the speed in which our search results load
What we did not do: plan for (or map out) our search experience.
Because of the content modeling, tagging and faceted search we built for the ASM site, we bypassed the out-of-the-box search tool in Kentico EMS (Lucene, based on SOLR). So we deployed Azure Search by Microsoft (note: the cognitive piece of Azure wasn’t ready in late ‘18/early ’19).
The heavily research-grounded approach we embraced for the website functionality was not mirrored in our search thinking. I naively assumed search would “just work” out of the box, not thinking about the organic evolution of the human search experience that had vastly changed in the last decade.
Search and Related Terms
Basic search indexers, untouched, typically parse content and rank it using an array that weights titles, URLs and keyword density in the body. You can usually sort these by freshness (publish date), most popular (most clicked) or relevancy.
You should seriously explore machine learning and indexing capabilities for your search approach to stay relevant and avoid adding frustration to your users search experience on your site. You should become familiar with the following terms:
- Federated search is when you integrate multiple sites/properties into your search tool, allowing for search of content across multiple platforms. Often, federated search is enabled through consuming multiple APIs from different sites or systems.
- Artificial intelligence (AI) is a “system’s ability to correctly interpret external data, to learn from such data and to use those learnings to achieve specific goals and tasks through flexible adaptation.” Whether it’s a self-driving car, search indexer, Alexa or the algorithms you experience as you traverse the web (think those creepy ads that follow you around, a.k.a. remarketing). That’s at the base level. Another definition? “Machines that respond to stimulation consistent with traditional responses from humans, given the human capacity for contemplation, judgment and intention.” Remember, though, that AI doesn’t have the human capacity for contemplation, judgment and intention.
- Natural language processing is a process by which programs parse articles, identify keywords/tags based on density, synonyms and other decision criteria to categorize the content based on what a machine already knows.
- My data scientist bestie says, “Think of NLP as having two subfields: Natural Language Understanding and Natural Language Generation. In NLU, you have classification tasks such as ‘Is this document about dogs?’ or ‘Is this tweet expressing a positive or negative sentiment? or ‘Is this e-mail spam?’ You also have document retrieval, which is what search is all about and is not a classification task. It’s more of a matching and ranking task: receive user query, retrieve documents most likely to match the query, rank according to some ranking criteria (which typically go a bit beyond relevance to the query and include market-usefulness for Google). Natural Language Generation is stuff like automated fake news writing, Alexa talking back to us, etc. Not super relevant to your topic.The overall point is that NLP is more broad and encompasses anything where a machine is expected to take text data as an input and/or produce text as an output.”
- Enterprise search is a catch-all term usually reserved for large, enterprise organizations. It catalogues all of their offerings—usually used by staff—and makes them findable using lots of other aspects (sometimes AI, sometimes NLP, sometimes federated).
- Personalization watches user behavior (clicks) to recognize patterns in preferences and serve up more personalized content to a person as they browse your site.
- Cognitive search, according to my favorite developer, “is, at the end of the day, mostly natural language processing combined with document and image detection.”
- Action-oriented search takes search design one step further. Rather than just listing a collection of results based on keywords, it tries to help a person complete a task. This is what Google does. When you search for a ramen joint today, you don’t just get a list of results that have the thickest keyword density of “ramen” & “restaurant.” Instead, you see an array of information with ramen restaurants closest to you, along with other relevant information such as hours of operation and phone number. Google understands that you are trying to eat ramen tonight, not just see a list of ramen restaurants. This is the basis for the catchphrase: search is a thing, not a string.
- Faceting (or filtering) allows your users to parse through large sets of information thanks to your grouping methods—doing some of the work for them. Think of the drill-down categories on Amazon when you’re shopping. You’ll search a product, then be able to hone in by price, brand, size, color, etc.
- Machine learning is a subset of AI where humans train a program how to learn or think, often programmers hard-coding rules into programs. It’s basically using algorithms to tell you something about something.
- The data scientist bestie clarifies: “In machine learning, [data scientists] hard-code a set of rules that guide the program to make decisions about which other rules should actually be executed in practice.Think of it as automating the work of programmers. It’s generally much simpler, cheaper and all-around better to let a human programmer do programming work, which is why machine learning is reserved only for cases where the rules are too complex or too numerous for a human mind to deal with directly. And the way a machine learns is actually fairly simple; it’s just doing guess-and-check really, really fast. It guesses a set of rules, looks at the training data, checks how close it got to the right answer, then nudges the next guess in a closer direction. This is done many, many times until the guesses start being close enough for a data scientist to be satisfied.
- Predictive search was introduced by Google, and it offers up suggestions as a user starts to enter characters into the search bar. These suggestions are typically based on behavior or what content matches in a database.
- Knowledge management. The official definition of knowledge management is “the process of capturing, distributing and effectively using knowledge,” which dates back to 1994 during the rise of the internet.
Do You Even Need On-Site Search?
- have a matrix grading system,
- ask them what their value proposition is and ask what their foundational, strategic approach to search looks like, and
- understand that it’ll be hard to get a real demo unless you are able to get the vendors plugged into your systems so that they can actually index your content. It’s worth taking the extra steps to sweet talk your IT team into giving these vendors preliminary access so you can see how search will really work for you.
Most importantly, ask yourself and your organization if it’s really even worth it to invest in on-site search. When I go to conferences now, I make a beeline for engineers who work in search as soon as I walk in. My newest engineer friend recently said to me: “The reality is that major search engines have spoiled us, and the commercially available tools feel lacking when compared to Google’s search experience (although, [stakeholders don’t] like it when you point out that they have 10,000+ engineers working on search every day).”
The 10 Commandments of Search
The major takeaways I can offer after diving into modern search in 2019 are:
- Search is a product, not a project. Your organization needs to understand that–just like websites, professional development or journals–search is an ongoing commitment that requires staffing and budget.
- Knowledge graphs. Use them (well, try to).
- Search is based on user’s needs, not your content supply.
- Start with the why (super appropriate for everything you do ever).
- You will need a team. If you’re taking search seriously at your organization, you need at least 1 full-time employee to take care of it. (Most speakers I talked to had a team of 4 during maintenance and teams in the double digits when deploying or enhancing search.)
- Be honest. Is it worth investing in search? Is it a good or bad thing if users are spending more time in search?
- Think of search as a thing, not a string.
- Search Experience Optimization: live it, learn it, love it.
- Look at metrics, reports. Survey your users on their search experience with you.
- Use diagrams to educate your stakeholders. They need to understand what search is in order to fund it.
RIP to the old you. Good luck.